Week 11
The meeting was postponed by two days due to personal reasons.
Finished
Added entering and scrolling result window
The fundamental function to have a better display of generating results.
If the video doesn’t play, please try this link: https://drive.google.com/file/d/1vKdb6dCexjZJ8cyl7zawE8xxWnPI5hIo/view?usp=sharing
Made prompt templates
By making these prompt templates, we are able to take any number of frame names and embed them into prompts, like filling the blanks.
Supported prompt templates:
- Zero-shot prompting
- One-shot prompting
- Few-shot prompting
- Chain of Thought (CoT) prompting
- Rhetorical options
Contemplating evaluation system
As is mentioned in the previous meetings, we are intending to implement an evaluation system, so that we can gather and store the feedback from users about the frame blending results. These evaluation information could be used for future analysing and training.
Currently, I plan to use files to store JSON-format evaluation data, since we are working on the HPC cluster and a persistent database might not be a good choice. Here is a JSON schema design by me for storing evaluation data. Parameters related to the generating process are to be stored, such as
frame namesprompt typesblending resultevaluations
Particularly, for evaluations section, we will have the users input feedback from different dimensions,
including
completenessclarityrelevancedepth of understandingcoherenceexecute timeadditional notes
Other Messages From The Weekly Meeting
Suzie pointed out my evaluation dimensions may need reference to prove the rigor of evaluation matrix, like how there dimensions are selected and what their connections and uniquenesses are.
It is correct indeed that we take rigor in every step of doing research and building systems. Therefore, we will continue to work on reading papers about evaluation and creating effective evaluation matrix for our frame blender.
To-do List For Next Week
- Continue with evaluation system, read papers and find effective evaluation matrix.
- Start doing documentation for frame blender.