Path

This page was located at docs/notes/menu/scouting/Introduction.md

Logistics Award - Scouting Palooza

Our Vision

Our goal in developing Scoutify is to simplify the scouting system into a seamless, user-friendly experience our team will enjoy. We want this experience to be unified, and include the most up-to-date schedule and information. We also recognize that this system needs to be flexible so it can be changed even during competition.

Our idea is straightforward. We utilize just the scout's phone and an internet connection. We also include the game schedule, real-time rankings (from The Blue Alliance), and pit scouting data to make this app useful for the entire team. With that in mind, we believe the web-based app Scoutify is an asset for our team to use in competitions.

Introducing Scoutify

Our web app, with the function of each screen shown in the flowchart above, is divided into 6 main screens visible to everyone, and an administrator screen for the scouting supervisors. There are several accounts with admin privileges, and other accounts with user privileges. Admins can assign users to scout for one of the six alliance-positions on the field. Once the admin presses "go" on a match displayed in our app, these scouters are automatically assigned to scout for the team in their field position. The benefit of this method is that each scouter is assigned the same area of the field to look at across matches. Meanwhile, admins can see when scouters submit the data, so they know when to switch to the next match in our app.

Additionally, while only assigned scouts can submit official data (quantitative data), any team member can attach comments to robots (qualitative data) to be considered during alliance selection. Finally, we display the Match Schedule - along with identification of our team's matches - for everyone to use.

Our app includes 5 tools to analyze the data we collect: a section with predictive results of future games, a selection of graphs to compare teams, a personalized chart for individual team performance, match-specific data, and a list of available teams sorted by different metrics. These tools, alongside others, are organized into different pages, or "screens" in our app, as discussed in the following sections.

Viewing Matches

The most fundamental part of data collection is knowing what match you're on. In Scoutify, any user is able to view the game schedule of the event 695 is competing in using the Match Listing screen. This screen displays all the matches in the competition, and the match we're currently scouting is highlighted in yellow. Users automatically jump to the playing match when they enter the website or switch to the Match Listing screen. Only admins can see the start and stop buttons, which permit them to select the match. This screen automatically updates on every device when a match is started and stopped.

Additionally, admins can click on any match to see who has saved their data for the match before continuing, which will be discussed further in the next section.

(Data is from the Daly division in the World Championship)

Collecting Data

In our app, collecting data is made easy and intuitive through the Data Collection screen, which scouters are assigned to by admins in the Admin Screen. This screen is programmed to be easily customizable and modular, as critical parts of data collection have to be updated every year. We employ a few fundamental types of user input that should stay constant year-to-year: numerical input, true/false input, and multiple choice selection. This year, we used numerical input to determine the number of game pieces picked up, boolean input to mark a preloaded robot, and multiple choice input to assess the docking performance of the team (shown in visualization). Additional qualitative assessment is available through an optional comment feature which is also accessible to non-scouters through the 'Comments' tab within the screen.

This year, we added a game-specific section for marking cones and cubes placed on the Charged-Up grid. This section has a grid which allows the user to mark that their Robot placed a cone or a cube by tapping on the corresponding cell in the grid. We have two of these grids for the Auton and Teleop periods of the game. Cells filled during Auton will be grayed out in the Teleoperated period of the game, so the scouter can't accidentally place a piece twice in the same node on the grid. Refer to the screenshots for a visual:

[Data Collection Form]

Organization and Displaying Data

Our app is organized into screens for different criteria. Our data visualization takes place in four of those: Team Summary, Team Details, Match Strategy, and Alliance Selector. Team Summary provides a selection of scatter plots and bar charts that compare FRC Rank (from Blue Alliance - via their API), Game Score, Charge Station, Links, and Num. Games Played. We mainly use this screen to get a sense of where our team stands in the competition. Also, if one team catches our eye, we can click on it to go to Team Details, where we can find the individual statistics and comments for each match the team played. This is important because some teams' robots break during matches and score zero points, which may affect our calculation during alliance selection. Meanwhile, if we just need a briefing for the next match, we can enter the Match Strategy screen. This screen provides information about each team and a predicted outcome for the match. We've used this screen after the first day of qualification to find what matches we have to win to seed high during alliance selection. Additionally, this screen allows us to determine when we are outmatched and should just aim for getting the 2 Ranking Points available to us. Our drive team also uses this screen to get an idea of what their next game will look like before they play it. Finally, we reserve the Alliance Selector screen for alliance selection itself. As teams are picked, our Admin accounts input the selections into an admin-only screen. Meanwhile, the Alliance Selector screen acts as a dynamic pick-list, updating in real-time as teams are selected. We do this by ranking each team in terms of its game score, charge station score, and links. We then weigh each aspect of their performance by a certain amount to get our sorted list. We are now working on a more adaptive algorithm that takes into account what capabilities existing alliances have when ranking possible picks. We are also changing the algorithm to prioritize teams that score differently from us (cubes vs cones, low vs high) to minimize overlaps in scoring.

Incorporating Pit Scouting Data

In addition to in-game data, our app also incorporates data we collect from the pits. We use a google form questionnaire to collect pit scouting data into a google sheet. We then send our scouts in groups of two to collect data from teams. When pit scouting, we encourage our students to hold a conversation with other teams, but we mainly care about robot weight, drivetrain, wheel type, and dimensions. We also have our scouts request to take a picture of the robot's drivetrain and its wheels to upload to the google form.

Once the data is collected, we upload it to a table in our database, where it is displayed on the Team Details screen. In addition to the pictures students may have taken, we scrape The Blue Alliance website for official images to make the team as recognizable as possible.

Ease of Collaboration

We built Scoutify as a web-based app specifically so it would be easy to give outside teams access. During the South Florida regional, we collaborated with Team 1523, Mega Awesome Robotics Systems (MARS) to help with data entry, and in a single online meeting we could explain how to use the app. We also shared our app twice with Team 2399 The Fighting Unicorns throughout the season. When we share our scouting with other teams, we make sure it's a win-win situation. Both teams get access to additional data, and the other team receives the same privileges we receive from Scoutify. We do this because we find that collaboration with other teams is a path to mutual gain. Not only does it free up half our scouters, but it creates an environment of positive collaboration that extends beyond scouting.

Responsive Design

We built Scoutify as a web-app for multi platform compatibility. Users can access our app through their phone, tablet, or laptop simply by entering our website url in their browser. The screen automatically resizes to fit the display, allowing us to display our app on big screens when presenting to teams or judges.

Applications During Alliance Selection

A good robot can acquire a top-ranking position, but advancing from there requires a solid, balanced alliance. FRC makes this interesting because qualification matchups are completely random, therefore rank is not always an indicator of robot quality. One of the best examples of this was the World's Championship last year. We were graciously selected by the 4th-ranked team as their first pick. However, they were lacking scouting data, so the next picks were up to us. Our information showed that team 5712, Hemlock's Gray Matter, had a low rank but a high game score (as shown in image to the left). Selecting them as our alliance's second pick proved to be a smart decision, as they performed exceptionally well during playoffs. The scouting process was extremely important in making this decision as it is often difficult to identify low-ranking teams that would be a good pick if we were only going off of the empirical data.

Dealing with Bad Data and Scouter Satisfaction

Dealing with errors and inaccuracies is one of the most difficult aspects of real-life data. However, we use several methods to keep errors to a minimum. We run system testing a few days prior to the competition. After ascertaining that everything runs smoothly, we don't change any server or client code to prevent creating new bugs. During the competition phase, we assign a scouter to each alliance position, after which we implement a few strategies to prevent bad data or lazy scouting.

Our first method is to make scouting as easy and enjoyable as possible to encourage accuracy. We arrive at the stadium early to secure a good view of the field. Snacks are provided for the whole team, but are placed next to the scouts to encourage people to stay in the area, and if anyone needs to take a break, admins can easily select an available scouter to collect data for the time being. Additionally, our data visualizations are available for everyone to see on our website, so scouters can see the fruits of their labor. In fact, our website credits scouters for the matches they scout. Our pit crew can also see how our robot is doing in real-time, contributing to a team appreciation for the role scouters play during competition. Finally, we hold a discussion with the scouters before alliance selection and ask them if any teams caught their eye. Not only is it crucial for scouters to know their time is useful, but they are the only members of the team that have been looking at robots the entire event.

Our second method seeks to acquire a metric for the quality of our data. We are exploring assigning a seventh scouter to a random duplicate team. The data from the duplicate team will not be added to the database, but rather checked against the data previously submitted for it. Using this data, we can assess individual scouter performance as well as overall data accuracy.

Our final method is simple. If we see any obviously inaccurate data, we have our scouters go back and correct the data in that match. This is usually not a problem since our app saves what they entered locally, so they can correct any misinputs and resubmit. If not possible for whatever reason, we can go into the database and erase the erroneous data ourselves so it does not affect the team's statistics.

Dealing With a Lack of Connectivity

Our scouters rely on their phones' data signals for internet connectivity. We switch phones as needed to guarantee that 6 scouters have internet at a time. If we are dealing with a lack of scouters, our app can function with as little as one scouter, prioritizing scouting unknown teams over teams we already have data for. We did notice that at IRI we had no internet connectivity at all. As a backup, we are exploring using a cellular modem and team laptops that connect via Ethernet. The router's antenna picks up the LTE signal much more effectively than a phone can, and Ethernet connects the laptops in a way that doesn't interfere with field signals.

Changes On The Fly

While we prepare for competitions as best we can, sometimes changing parts of our scouting system at competition is unavoidable. Simple changes like adding users can be done from within the app. We have a list of usernames and passwords we can quickly give out. Bugs, if found, are quickly fixed by our team and pushed to the server. Since Scoutify is web-based, the changes take place immediately. Finally, unforeseen issues with data have to be taken care of from within the database. We organize different elements of data into different tables in our MySQL database. Therefore, it's pretty easy to go in and change data with a visual utility like DBeaver, the tool we use to access our database. In addition, we use a Virtual Private Server (VPS) for easier maintenance and better performance compared to a shared hosting service. We also implement HTTPS and WebSockets to secure the website and allow for features such as real-time match updates.

Our Impact Through Scouting

The development of Scoutify played an essential role in our team's success this past season. Students got to see a real-world demonstration of how data collection, where the team works as a cohesive unit, can shape decision-making and lead to a more informed, data-driven decision-making process.

Through the project, our students learned to write code as a team and apply it in high-stake situations. Additionally, the project opened up new avenues of collaboration with other teams through joint scouting and sharing information. Finally, our success using Scoutify led to a greater team appreciation for the advantages of scouting and fostered a sense of unity and collaboration throughout the entire 695 crew.

Our drive team says that "driving skill is important to win qualification matches, but the importance of scouting is what lets us win tournaments." In our team, we feel alliance selection is the most important 15 minutes of a competition weekend, and Scoutify was developed (and will be continuously improved) to support alliance selection. With this in mind, we will definitely continue to develop this project in 2024 and future seasons.

As we continue to develop, we would like to open up this scouting system to any FRC team that would like to use it. We are confident that other teams would find it as beneficial as we have.