Coding Period

Reporting my progress on coding period at GSoC21 right here. In the following article I will write my progress in a list format for each week, i.e. accomplished tasks and read more details part contains more details about the accomplished tasks and screenshots about them. As well as at the top of the article you can find two lists one for issues submitted during coding period and another is contains the submitted pull requests.

I hope you find it useful for you.

Table of Contents

• Important Links
• Terminologies
• Issues List
• Pull Requests List
• Phase 1
  • WEEK 01 ^{07 Jun. - 14 Jun.}
  • WEEK 02 ^{14 Jun. - 21 Jun.}
  • WEEK 03 ^{21 Jun. - 28 Jun.}
  • WEEK 04 ^{28 Jun. - 05 Jul.}
  • WEEK 05 ^{05 Jul. - 12 Jul.}
  • WEEK 06 ^{12 Jul. - 19 Jul.}
• Phase 2
  • WEEK 06 ^{12 Jul. - 19 Jul.}
  • WEEK 07 ^{19 Jul. - 26 Jul.}
  • WEEK 08 ^{26 Jul. - 02 Aug.}
  • WEEK 09 ^{02 Aug - 09 Aug.}
  • WEEK 10 ^{09 Aug - 16 Aug.}

---

Important links

• RA Organization Repo Link
• RA Forked Repo Link (check out branches)
• Blog Series Link

---

Terminologies

• F stands for feature, e.g. feature no.1 will be F01
• B stands for bug, e.g. bug no.1 will be B01
• PR stands for pull request, e.g. pull request no.1 will be PR01
• RA stands for Rapid Annotator
• VIA stands for Vgg Images Annotator

---

Issues list

1. Going to 'Settings' of a new experiment leads to an Internal Server Error

   Description

   When I go to a new experiment and click on its settings immediately without making any uploads to this experiment (empty experiment) it leads to unexpected behavior;
   "UnboundLocalError: local variable 'pngImageB64String' referenced before assignment"
   The logs and screenshot describes it more. It gets Internal Server Error on production mode, and in debug mode it is as the following.

   Screenshot

   

   Logs

   

2. Unexpected behavior when adding or removing annotators at an experiment

   Description

   When adding/removing annotators to an experiment in a some quick way (I did that just for testing the functionality but I found that), it leads to adding same user twice sometimes which is not logical. Is it alright or not very important to look for fixing it?

   Screenshots

   1 The behavior when using delete and add annotator functionalities.

   

   2 The delayed delete

   

3. Export results to sheet as CSV doesn't work

   Description

   Export results to sheet of an experiment works only as ".xlsx" files and "csv" dosen't work, it leads to unexpected behavior
   "UnboundLocalError: local variable 'app' referenced before assignment", screenshot is attached below shows that.
   I found that due to missing import as I discovered I will submit a pull request to solve that.

   Screenshots

   

4. View large experiments and results viewing take very long time

   Description

   Currently the loading time of viewing a large experiment is very big as Dr. Peter mentioned to me and Gulshan, I have reviewed the code base and found an issue in the pagination behavior it takes processing time for the whole data, while it is expected to be done only on the portion of data which will be displayed to the user as per page request. So I will work on that to optimize that time and make lazy-loading instead of the current.

---

Pull requests list

1. Fixed view settings bug of a new/empty experiment

   Description

   This pull request solves the issue of viewing settings of a new experiment

   Screenshot

   

2. Fixed adding/removing annotators bug

   Description

   This pull requests solves this issue Unexpected behavior when adding or removing annotators at an experiment #47
   It is a simple solution as I found the location reloading in unnecessary and making the unexpected behavior because immediate reloading makes the code in an inconsistent state and I didn't get what is their purpose so commenting them fixed that issue.

   Screenshots

   1. Firefox is working right now

   

   2. Making quick adding is working correctly as well as removing, everything is fine. (this on Google Chrome)

   

3. Fixed no progress alert showing when removing all annotators

   This pull request is a complementary to the latest pull request which is merged I discovered there is a problem when removing all annotators, the alert was causing some problems so I have solved it.
   

4. Fixed export results to csv issue

   Description

   This pull request solves Export results to sheet as CSV doesn't work #50

   Screenshot

   

5. Fixed loading time of viewing an experiment issue

   Description

   This pull request solves View large experiments and results viewing take very long time #51
   

   Screenshot

   Left side is after being fixed, right side is before.

   

6. Video Snippet Annotated column added to results exporting at concordance experiments

   Description

   Added a new column to the concordance experiments exporting results "Video Snippet Annotated" which is simply the same as Video Snippet but with start and end modified based on the experiment's display time [before_time and after_time]
   By applying the following equations

   new_start = video_snippet_start - exp.display_caption.before_time
   new_end = video_snippet_end + exp.display_caption.after_time

   and replacing those with the existing on the video snippet using regular expression.

   Screenshot of an experiment

   

7. View results filtered by annotation level and labels feature

   Description

   Added new two select boxes to allow viewing results filtered by annotation level and annotation level's labels.

   Screenshot

   

8. Reordering the annotation levels in a more easier way and more efficent

   Description

   Making the reordering of current existing annotation levels was a hard and need to open each annotation level to check what number is given for that and then keep all of them in your mind to make a new level or just change an existing annotation level 's level number so In this pull request proposing an easier way to do that as explained in the following screenshot.

   Screenshot

   

9. Multichoice feature branch merging to master branch

   The following features are added from multi-choice feature branch

   1. Allowing multi-choice annotation levels. i.e. (select multiple labels for same annotation level and add a field for each label as other text i.e. commenting on each selected label from the annotator)

   Multichoice annotation process and viewing results
   

   Adding a new level
   

   1. Export results at long-format (New exporting results format)
      

   2. Export results at wide-format (New exporting results format)
      

   3. Optimized the existing exporting results time

   This pull request contains database changes and new added package to apply them you need that

   # to install flask-migrate package
   pip3 install -r requirements.txt
   # Initalize the flask-migrate files.
   flask db init
   # Make a migration, it is like commit it and a comment beside.
   flask db migrate -m "Initial migration." # OR just add your comment right there  
   # Then apply your migration to the database, it is like push.
   flask db upgrade  
10. Continue a specific experiment using exported wide results format feature

    Description

    To allow experimenter modifying the results exported file (at wide format) and continue from the point they have edited their results to a new experiment. for example removing a row from the sheet (file)

    So the following changes has been needed to be added to wide format results export

    1. Added all attributes of file (file name, content, concordance line number; for concordance experiments and edge link and rest of them)
    2. Merged the concordance file data to the exported results wide format file (in case of it is a concordance uploaded type)

    So to continue an experiment you need to

    1. Specify which experiment to continue (select box based on your experiments)
    2. Upload the exported results in wide format
    3. Give it a new name (default: Copy of [old experiment name] [id])
    4. Give it a description (default: "")
       

    Effects are same as creating a new experiment based on the old experiment results file

    Screenshots

    1. Copy of image experiment
       

    2.Concordance experiment (it creates a new concordance file based on the given results file)
    

    3. Text experiment
    

11. New feature added and modifications on exporting results and solving view results with no annotators issue

    This pull request contains the following

    1. Solves the problem of viewing results of an experiment while it has no annotators

    this due to the hard coded id sent when len(annotators) = 0 at line 73 @RapidAnnotator-2.0/rapidannotator/modules/add_experiment/views.py

    Before

    

    After

    

    2. Allowing experiment's order to be modified between normal and random

    

    3. Disabled uploading once the experiment has already a done uploaded concordance file

    

    4. Exporting results of concordance at wide format will be only contains the edge link if it is new scape type

    

    Kindly review that and tell me if there are any further modifications needed

12. Annotations adds-on (coordinates feature)

    Description

    The following pull request adding is an adds-on to the current annotation process system to allow the experiment get more information about the data and use them with their process of information and features extraction and get more intuition about which part of the data is meant for this annotation

    Usage

    This feature meant to be used by the experimenter in the step of modeling and preprocessing.

    Screenshots

    1. Image experiments VIA Vgg images annotators, I have integrated the images part to Rapid annotator
       

    2. Audio/Video Built a time controller for selecting a time start and end of the video those controllers (buttons and ion-slider manages the video running duration and starting and ending time
       

    3. Text experiments extended the current base highlighting feature of concordance to be able to use on the normal experiment also
       

---

Phase 1

WEEK 01

1. Fixed graph reloading at the settings of an experiment .
2. Found a new bug happens when exporting the results as CSV.
3. Updated the setup of adding annotation level; to allow marking on multi choice level
4. Finished the needed UI functionalities at the annotation process for supporting the new multi-choice functionality.
5. Finished the required database schema changes for multi-choice functionality.
6. Edited the annotation process endpoint to adapt with the new request structure.
7. Keeping working on it and testing and finding bugs on the new added or existing code base.

---

WEEK 02

1. Optimized the viewing of an experiment loading time in terms of large experiments files I liked doing that xD
2. Exporting results after making the multi-choice functionality at long format.
3. Edited exporting results at the concordance exp. type, added new column of video snippet annotated changed, start and end times, offsets!

---

WEEK 03

1. Filtering view results based on a specific annotation level or annotation level label

---

WEEK 04

1. Added ordering the selection of label at annotation process (new column added at long-format)
2. Finished wide format at exporting results
3. Added warning when user changes the multi-choice option of an annotation level
4. Finished viewing a specific entry within the changes on multi-choice levels
5. Working on changing a specific file annotation from viewing results page
6. Working loading time of exporting an experiment results at standard format

---

WEEK 05

1. Allowed changing a specific file annotation from viewing results page
2. Optimized loading time of exporting an experiment results at standard format and wide format
3. Seeing the level order number of a particular annotation level (level number attr.) while setting an experiment and being able to change that number if needed in a way more efficient than we currently have.

Phase 2

---

WEEK 06

1. Re-uploading an experiment and continue to based on the results exported in wide format feature

---

WEEK 07

1. Fixed the bug of going to results when there is no annotator
2. Finished disabling uploading after first uploaded done on concordance
3. Added changing order of anexperiement
4. Changed display edge link for new scape only
5. Look at VIA and checked out if it could be integrated to RA or not

---

WEEK 08

1. Working on switch display order functionality (figure out its effect)
2. Working on the VIA(VGG-Images-Annotations) integration to allow new annotation abilities, i.e. adding polygon boxes and starting and ending of the videos Current UI updates integrated

---

WEEK 09

1. Support Highlight feature of text annotation experiment for non-concordance experiments
2. Working on adding UI components for the video and audio experiments
3. Going to edit the database to store the annotations as JSON column type

---

WEEK 10

Integrated VIA and added audio/video controllers for time range from scratch as well as extended the highlighting feature and fixed a lot of bugs while working on those tasks which I call the annotations adds-on and add it into the database, I would like to explain what structure has been followed

• Image experiments it contains an array of shapes which was drawn on the image
• Audio/Video experiments a json object contains starting time and ending time in seconds
• Text contains the first index of the selected character within the highlighted text

Lets see a demo of that working!

• Image

• Video/Audio

• Text

---

TODO

• Keeping documentation up to date. (Dr. Vera)
• Clustering on different types of data and clustering connectivity. (Dr. Peter)