RHoK 2.0 - Toronto/Project/Tweak the Tweet - user testing

Contents 1 Notes from Day 1 1.1 First Impressions 1.2 Testing Phase 1.3 Questions/Comments/Feedback/Issues (user testing perspective) 2 Potential solutions 3 Notes from Day 2 3.1 What we hope to accomplish today 3.2 Other Thoughts 3.3 Proposal for new RHoK/CrisisCommons Project 3.4 Next Steps

[edit] Notes from Day 1

RHoKTO User Testing of Tweak the Tweet [1].

[edit] First Impressions

We first approached using this application with minimal or little knowledge... Initially, we reviewed the wiki and the various links in the wiki Project Definition description and proceeded to get started with setting up the "dummy" twitter accounts so as to not overwhelm our regular twitter stream.

• LindaRHOK - Linda (Linda Ogbeide)
• stian_disaster - Stian (Stian Haklev)
• christine_rhok - Christine (Christine Crowley)
• katyrhoks - Kate (Katherine Came)

With minimal to little reading of the various links and documents, we proceeded to test random tweets assuming our disaster was a terrorist bombing attack of the Toronto Subway system (TTC). Consider this an attempt to familiarize before "going deep" and/or a simulation of real world conditions were trainig of TtT would be minimal.

We used the Twitter hashtags specified here [[2]] and looked to see if our tweets populated the following: (a) the google document[3] and the map [4]. Truth be told, we found it a little confusing at the beginning. This prompted us to connect with Catharine Starbird (not to be confused with Katherine Came) as soon as possible to better understand why were having initial trouble populating the outputs.

Through our Skype conversation with Catharine, we learned that (1) the program only runs locally on Catharine's computer, which means that she has to set up collection for any event, (2) that the program is too complicated to easily give to someone else to run on their own, and (3) that the current outputs (a public spreadsheet and map) require people who want to consume the data to learn the Google APIs - ideally, we would provide an output of a GeoRSS feed (requested by Sahana) or some other data standard.

Stian was very interested at this point in better understanding the code (what makes a tweet successfully processed by TtT program) and worked closely with Catharine to obtain it and started taking a deeper look. He learned that the original code is in Tweepy and is under a Creative Commons license (a fork of that project - or would somehow extend that license). Stian reasoned that (1) be rewritten in a modular form in ruby (decouple the different functions of the program into different modules); (2) the code should be made public (get the entire code base into a form where it could easily be downloaded from github and run by a new user and easily hacked on by people), (3) remove the hashtag definition define the hashtags in a config file, without hardcoding in the logic, so that it is easily editable by a non-coder (e.g., a NGO).

Additional notes:

While Stian took a deep-dive into the code, the rest of us decided to continue to test the application and document issues/problems we came up with...

[edit] Testing Phase

As a group we decided that the black box nature of the program required some sort of systematic and routine testing using first primary tags alone, then data tags alone, then combinations of the different tags together to better understand how the program was handling or processing tweets. Here are some results:

#TtT_Test alone does not seem to do anything except populate the Twitter stream. It needs to have a primary tag (see definition of primary tag here[[5]].

Primary Tags. In order to populate the spreadsheet, all tweets seem to need the #TtT_Test and the #primary tag. We ran a test combining a primary tag with #TtT_Test E.g. "#ttt_tweets #road Front St" (Note. Instructions say that directly after the primary tag, write who is okay or missing, what is needed, offered, damaged, etc. or the details of the damage, etc.. Here are the results (Yes = tweet populated the sheet; No = tweet did not populate the sheet):

#imok --> YES (Note. #iamok works too)

#ruok --> YES

#missing --> YES

#trapped --> YES

#injured --> YES

#injury --> YES

#need --> YES

#offer --> YES

#road --> NO

#traffic --> YES

#closed --> YES

#shelter --> YES

#damage --> YES

#facility --> YES

#service --> YES

#power --> YES

#open --> YES

#closed --> YES

#cancelled --> NO

Feedback: We are wondering why some tweets do not populate the spreadsheet when using #TtT_TEST with the following primary tags #road, and #cancelled. Perhaps these should be data tags? Also, some of these could provide meaning without additional information,e.g., "#TtT_Test #imok --> indicates that the twitter user is ok. We also found that when using two primary hashtags in one tweet, the program only captures or categorizes the tweet as one but not both primary tags. One of the builds is to ensure that a record gets created for each primary tag (though the tweet would be dublicated).

Data Tags. We ran a test combining the #TtT_TEST and the data tag only without a primary tag. None of the tweets populate the spreadsheet when using the data tag without a primary tag. Hence, must use a data tag with a primary tag. Also, when using data tags, these are useless from a semantic point of view without providing additional information following and qualifying the tag. Hence the need to make sure that the these tags are followed by qualifying information. We ran a matrix that combined variations of the use of primary tags with data tags (see list below) and had mixed results (results not shown).

#loc location -->

#src source of the information -->

#contact whom to contact -->

#status status of person or situation -->

#details more info -->

#info more info -->

#time when original report occurred if not happening now -->

#date when original report occurred if not happening now -->

#number of injured, or shelter spots open, etc -->

#capacity of injured, or shelter spots open, etc -->

Feedback:

• We are wondering why the tweet syntax is so complicated? For example, why the data tag must be used with the primary tag (we later learned that the primary tag actually triggers the TtT program); overall the tweet syntax becomes complicated, i.e., event tag + primary tag + data tag plus qualifying information.
• We can't help but think that this would be very difficult to know in general (for the average or lay twitter user) and even to learn in a crisis situation, especially for the casual twitter user.
• Also, there is some confusion with the #loc tag. We are particularly stumped by the following instruction "if there's no text after the primary tag and no location info, the tweet will not be processed" - not sure if this is for the spreadsheet or for the map.
• Also, if a user tweeted geolocation data using the twitter geolocation feature, would one still need the #loc tag? It seems redundant.

Geo-location Information. The instructions were to include the #loc tag w/ textual description of the location and either enable automatic geo-locating or include lat, long if possible in this form
Feedback:

• We had the most trouble with geolocating tweets.
• In general, general location tweets did not populate the map. The only tweet that appeared on the map (initially) was probably because it had geolocation information: "#ttt_test #service is out #loc between Bathurst and Spadina 43.648491,-79.404489" [6].
• Long/lat info requires an online program such as [[7]], which is impractical during a crisis, especially from a mobile device; unless one uses the geolocation function in Twitter on desktop or mobile device, but not everyone might use this.
• It seems important to note that someone won't automatically think to use the #loc hashtag so that the information gets parsed into the document or gets onto a map.
• There appear to be differences in how location information gets represented. For example, in urban centre, some may identify locations with buildings (e.g., Eaton Centre, UofT, AGO) whereas in rural or residential areas, the location information would be an address.
• We are not sure whether or not the lat/long information is being properly located on the map (e.g., the College Park tweet is located at Yonge/Davisville instead of Yonge/College).
• It turns out that we were a bit confused with how the #imok and #ruok tags are classified as #missing in the map.
• Given the critical importance of address information during a crisis, it seems counterintive that address/location information be the most complicated aspect of the program.

Note. Stian is adding a level of code that uses a Google api to transform address information into a long lat, ideally with a #loc or #loc-type hashtag. Regarding procesing the geolocation data, there are so many different transforms. It might be possible to pack all of this in its own piece, and create a test suite for it (with a ton of different ways of formatting locations). That would be really valuable - and could potentially be useful for other projects out there too. Also, Catherine indicated that Sahana has asked for the info in GeoRSS form [[8]]

[edit] Questions/Comments/Feedback/Issues (user testing perspective)

• Through communication with Catharine, we realized that some of our initial confusion was due to the fact that we didn't realize that the program was not initially "turned on", that there was a delay of 20 min to populate the spreadsheet [9] and the map [10]. This is problematic for testing how the program processes the tweets in a timely manner.
• During our testing phase, some of our test tweets had been deleted from the spreadsheet, making it rather difficult to test the success of the tweets.
• Twitter would consider our test tweets as spam and shut down the account, forcing us to create new ones. This slows down the testing process. It also turns out that in boulder fires, workers had to keep opening new accounts to keep process going - innefficent during a crisist.
• Testing the program/algorithm via Twitter is not efficient overall. This testing process can be done outside of Twitter using an independent module to refine the coding and then user test the resulting final algorithm with Twitter.
• Initially we did not know how anyone would know what the event hashtag would be. For this test, we are using #TtT_Test because it is assigned, but how would people know which hashtag to use in the event for a real disaster? We thought, do we need to create the hashtag to identify the disaster and then notify the public? We later found out that the event hashtag gets determined organically based on whatever the Twitterverse is using as a hashtag for the crisis in questions.
• More importantly, we don't know how people would just know what the primary and data tags would be? Let alone the syntax of the tweet. Per our skype conversations with Catharine, we found out that the syntax of the tweet is "Event Tag + Primary Tag (qualifying information) + Data Tag (qualifying information)", where the data tag and qualifying information would be optional. We can't help but think the cognitive load of this information would be too great a burden for the lay user, while an emergency worker could be trained to know this information.
• We also did not immediately appreciate why there two categories of tags? Primary and data tags? Why can't all the tags be treated as primary tags? We found out that the primary tag is how we classify a tweet. Each tweet can only have one classification. It is either a missing person's report or a need or a damage report, etc. This allows workers/users to send needs to one kind of system and missing persons reports to another - and can help us separate different types of reports on the map. Also, the primary tag is used as a filter - this is how to figure out that people are using TtT - when they include the Event tag and a TtT primary tag, they are usually (90%) trying to use TtT. The thought is that many tweets would contain several secondary tags - contact info + location + source etc. THe program didn't initially have this distinction but they felt forced to make this distinction to be able to better classify the tweets. One major purpose of the syntax is to reduce ambiguity in the text - so it can be automatically classified with near 100% accuracy and little (<5%) noise.
• Education becomes an important need in this context, not only how to format a tweet, but also rescue workers would need to have a priori knowledge of both the maps and the google docs link. We all agree that without special training, it is not intuitive how to properly syntax a tweet to make the program work... The cognitive load is high. People would need to know the syntax of the tweet. Perhaps need a awareness campaign. Some notes address this point here [11]
• We later also found out that the intention would be to have rescue workers or disaster relief workers would train users/twitterverse on how to set up the syntax of the tweet during an event. We believe this happened during the Boulder fire.
• An interesting point about this program is that why one needs to use the foregoing syntax is in some ways more difficult to explain than how to create the tweet with the proper syntax

[edit] Potential solutions

1. Build a simpler system for the lay public (New) or an add algorithm (in the form of another module) to the existing program that can process non hashtag heavy tweets (e.g., use a Bayesian analysis to process the tweets). This is to help accomodate the need for non-instruction heavy program. (New solution)
2. Educate audience (e.g., rescue workers / disaster relief workers) on this system and have them educate public during a crisis on how to syntax the tweet (Existing solution)
3. Have a social awareness program for the lay public (existing solution).
4. Create simple instructions (PSAs) for tweeting emergency info that can be shared with local media? They can broadcast this information to listeners/followers to spread the information out and help build better emergency tweets. Use a PSA.
5. Have an easier way to not only share address information but also to geolocate (or enable geolocation of) the tweets (existing solution).
6. Create / force / encourage widespread use of the following web application (editor): http://epic.cs.colorado.edu/TtT/editors/haiti/. This is currently a training tool: one ticks check boxes and radio buttons and the editor creates a tweet. It does not push the tweet out to twitter (one needs to copy/paste to Twitter), but it can potentially be programmed to send the constructed message to Twitter. (New solution) It is important to note that the syntax in that editor is outdated and does take into consideration the current syntax/coding.
7. It would also be interesting to take the former solution and turn it into a mobile app so that the lay person could access in a crisis and would not have to learn or know what hashtags to use. This has been talked about alot in CO and at CrisisCommons, but no resources have yet been assigned to this project.
8. One could also have an IVR program (Interactive voice response system) for the phone that could process information from callers using 1-9 keypad numbers in such a manner that depending on persons selections using the keypad, a tweet is created that can be processed by TtT.
9. There is also opportunity to have this program in different languages.

[edit] Notes from Day 2

[edit] What we hope to accomplish today

• Build a new set of user testing instructions. It is possible that some of the initional confusion could have been avoided if we had studied this document [12] a little more closely because we really did not appreciate the extent to which the structure and syntax is important to the program. Some early "issues" (e.g., lack of successful population of the spreadsheet and maps) did not turn out to be issues with the program per se, insofar as with the proper syntax, the application works beautifully (see exceptions). On the other hand, we felt that there was too much information across too many different web pages and documents to efficiently review the information and use in a timely manner or even know that the go-to document for this RHoK #2 user-testing exercise was that document. As a result, we feel that we need to build a simple document that better advises the test user on what to read and what to do first.
• Continue to work on Coding: The program is on github and so is shareable. The program has been recoded into Ruby modules and currently it 'listens' to the twitter feed and begins to parse and outputs the debug information about how it's parsing. Stian plans to put a program on the internet whereby one enters a tweet in a text box and it outputs how the program is parsing the information.
• Presentation: You can view a presentation of the past 2 days here. See Slideshare presentation here:[[13]]

[edit] Other Thoughts

• Twitter is a very unstructured way to communicate. At it's most basic level, anyone who has an account can post information without any formal syntax and it seems counterinterintive that one should use an informal communication channel for such formal/structured tweets. The preponderance of early unsuccessful tweets in our early experience with this program speaks to the realities of 'real world' conditions and the need/obstacles/implications of needed to educate on the syntax for a tweet to be successful.
• Some other programs that were mentioned in the context of this project were as follows:

1. Ushahidi [[14]]
2. Crowdflower [[15]]
3. Pakreport [[16]]

• Some of us wondered how the program could be tweaked to become a health related app.

[edit] Proposal for new RHoK/CrisisCommons Project

Add additional component to program to analyze user tweets that do not follow the prescribed syntax but that still get processed by the TtT program and be captured in the map and spreadsheet. Rethink what the algorithm should be to capture the tweets during a crisis. There is opportunity for the following folks to communicate with each other regarding their programs:

1. Need Help [17]
2. I need somebody [18]
3. IAmNotOk [19]
4. ImOk [20]
5. We are Helping [21]
6. Tweak the Tweet

[edit] Next Steps

Non coders

• Generate text files with lots of different formats of geolocation data populate a txt file. Can obtain tweets from previous events/mapps

Coders

• Work on geolocation normalizer
• Write parser with keywords
• Logic for the sparser - Need to know Ruby (Not Ruby on Rails)