archimg.com — Collected works by Stephen R. Mallon, RA

Music Videos...

kept me very busy during the pandemic.

The videos listed below can be found on YouTube, either on my channel or on the Cordus Mundi channel.

Music videos are best experienced with stereo headphones or good speakers.

Back to Top

History

The Spring of 2020 was one of the most beautiful I can remember. Although the COVID-19 pandemic was breathing threats of destruction, nature was getting a much-needed sabbatical. I was grateful to live in an area where I could take long outdoor walks and pay attention to the emergence of Spring in ways I hadn't taken time for in previous years. The air was fresher. The sky was clearer. There were no airplanes droning overhead. The ubiquitous noise of tires on the pavement of nearby roads was silenced. (Could we make these conditions part of the new normal?)

Along with Spring, my neighborhood came to life. Neighbors I'd never met before were forced to stay home and were out taking walks. I got to know the dog-walkers and their dogs, all from safe social distance of course. In the past, the only times I had felt this kind of rapport with my neighbors were when we shoveled ourselves and each other out from blizzards. Now we were experiencing community in beautiful weather, albeit with masks.

But one avenue for building community was completely shut down; singing in public. Frightening reports had labeled choir rehearsals, concerts, church services and other occasions where people sing together as “super-spreader” events. Concerts I was rehearsing for had to be abruptly cancelled. Concert seasons were initially put on hold, and then shut down until further notice. While adapting to other privations of the pandemic, I found that not being able to prepare music for live performance left a gaping hole in the rhythm of my life. Along with other musicians, I had to go virtual with my desire to make music.

Grids of singing faces started showing up online – with varying degrees of success in synchronization and sound quality. I started doing research. What technologies and techniques were others using to produce virtual choir videos? The latency of Zoom made it unsuitable for participatory singing. Low latency apps such as JackTrip worked well for small groups of tech-savvy musicians, but larger groups of potential participants wouldn't have the required high speed internet connections and interface requirements would scare off technophobes. I looked for techniques with an easy entry point for participants using readily available technology. Selfie videos using a cellphone camera app looked like the best way forward, even if processing these would involve a lot of editing work.

Experiments during March 2020 resulted in my first video, “Now is the month of staying” where I sang all the parts myself. This led to subsequent videos for Cordus Mundi and other groups. My earliest videos were assembled using Shotcut, a free open source video editor which fit my budget and served my initial purposes admirably. As video concepts became more involved, I needed a video editor with more features and chose Adobe Premiere Pro. In Cubase (the digital audio workstation I've been using for years) I had a familiar interface with all the tools I needed for sound editing. Over the course of producing several videos, I came up with the Production Process listed below.

Back to Top

Production Process

MUSIC PREPARATION

  1. Scan sheet music to PDF format and convert to MIDI format.
    (software used: Neuratron PhotoScore)
  2. Confirm tempos, dynamics, cutoffs, breathing and other markings with Music Director.
  3. Create an audio tempo track from MIDI file, reflecting markings from Music Director. Adjust tempos and trim note values as needed for cutoffs, breathing and phrasing.
    (software used: Cubase)
  4. In addition to a condensation of all MIDI parts rendered with piano samples, make a different version of the tempo track for each voice part so that singers can hear their voice part reinforced in the audio mix with a sampled instrument sound. A starting pitch and at least one measure of metronome clicks are given before the start of the piece. In some cases, quiet metronome clicks can be added to the entire tempo track with beats subdivided where necessary to make tempo changes easier to follow. Too much metronome should be avoided so that singers aren't distracted from focusing on phrasing.
    (software used: Cubase)
  5. Send MP3 files of audio tempo tracks to Music Director for approval. Make adjustments to tempos and phrasing if necessary. No tempo changes can be made from this point forward in the process.

RECORDING PROCESS

  1. Send MP3 files of audio tempo tracks to singers. Singers can choose to use piano track alone or piano track with their voice part emphasized.
  2. Distribute copies of marked score to all singers along with any interpretation instructions from the Music Director.
  3. Send self-recording instructions to singers along with audio tempo tracks. Recommendations include dry room acoustic, simple background, camera placement/orientation, adequate face lighting and eye contact with the camera. Singers can be encouraged to memorize as much of the score as possible to facilitate good eye contact. Ask singers to use small unobtrusive earbuds if possible rather than big clunky headphones.
  4. Singers record singing selfies using the video app on their cell phones while listening to the tempo track on earbuds.
  5. Singers post their videos to a file sharing app (such as Dropbox) in either MP4 or MOV format.
    (preferred resolution 720P in MP4 format)

AUDIO EDITING PROCESS

  1. Download videos posted by singers and extract audio to MP3 format by playing each video and recording sound output to system audio.
    (software used for this and following steps: Cubase)
  2. Import extracted audio MP3s into DAW (Digital Audio Workstation) and align them with the original MIDI tempo track. This can be the most time consuming part. Even if singers sing at the same tempo, consonant sounds are not always synchronized the way singers would naturally do in an ensemble. Most consonant shapes can be visually identified in the wave forms but must be confirmed by listening carefully to each audio segment. Clip and stretch audio segments to align with tempo track at a resolution of the nearest 128th note. In order to have vowels sound on the beat, consonant waveforms should be aligned to fall into the "heel space" of about a 64th note before the beat. This means going through each audio file syllable by syllable. Scoops and divergent vowel sounds can't be so easily fixed. In most cases it is necessary to mute the offending singer for the duration of the infraction. In some cases it is possible to find the correct vowel sound from another syllable and paste it in place to fill the void. This is particularly necessary in small ensembles for singers whose distinctive timbres become noticeable by their absence when muted. Not all singers are able to breathe at the same times so premature gasps for air can be edited out. Other noises such as throat clearing and page turning sounds should be edited out.
  3. Once timing of consonants is synchronized, singers are grouped by voice part into sections to balance voices of varying timbres for the most unified section sound possible. During this process, fix intonation (and occasionally wrong notes) by pitch shifting. Where necessary, pair groups of singers and listen for beat notes for pitch fine tuning to the nearest cent. Once all singers in the same voice part are properly mixed and tuned, assign them to a group channel for balancing sections in the overall mix.
  4. After group channels have been assigned for all voice parts, check and adjust balance for best overall choral sound. Use panning controls to position sections left or right to match their placement in the video.
  5. Automate variable reverb controls in the sound mix to correspond with spaces depicted in the video. For example, in Cordus Mundi's video of Biebl's Ave Maria, the reverb settings for the beginning of the video corresponded to the acoustic of a medium sized church. To give an outdoor ambiance to tree sequences in the video, I used a shorter reverb time with a longer, but more subdued first reflection. For high and narrow cathedral spaces I used a much longer reverb time with a shorter first reflection time. Cubase has a convolution reverb feature called RoomWorks that allows for many variations in reverb settings.
  6. When the audio sounds right, export the master sound track in MP3 format. The master sound track will be imported into video production software as a basis for timing of all other audio and visual assets.
  7. Send an MP3 file of the audio sound track to the Music Director for approval. Any changes in phrasing cannot be made after this point. General discussion of video concept possibilities with the Music Director might be appropriate at this point, but don't commit to a presentation strategy that would be problematic or difficult to execute.

VIDEO CONCEPT

  1. To avoid a static grid of singing faces (or gimmicky transitions to relieve the monotony of such a grid) develop a video concept that can tell a visual story to enhance the music. The video concept should choreograph placement and movement of singers on a stage set of background images or videos selected to enhance the lyrics and feel of the music. When singers will be moving during the course of the video, it is helpful to place them within concept-appropriate containers such as globes, TVs, bubbles, balloons, cars, lanterns, space suits, etc. Image assets for these containers should be prepared ahead of time.

PREPARATION OF VIDEO ASSETS

  1. It is helpful to prepare grids or container diagrams for each scene in the video. These images should be sized to match the video output resolution (such as 1080p – 1920px by 1080px) and can be imported into the video editing software for use as background templates to aid in precise location and movement of singers and containers.
    (software used: Photoshop)
  2. Assemble visual assets from royalty free photo and video sites (such as Wikimedia Commons, pexels.com, unsplash.com, etc.) Decide which assets to use and prepare them using image editing software to adjust size, resolution, cropping, contrast, brightness, saturation, etc. To expedite production schedule, image assets can be prepared while waiting for singers to record and submit their videos.
    (software used: Photoshop)
  3. Collect royalty free sound effect files and edit/trim them using sound editing software. Sound effect audio files can be enhanced with reverb and other effects as needed to match the master sound track.
    (software used: Cubase)

VIDEO EDITING PROCESS

  1. After importing the master sound track into video editing software, import original singer videos (including sound) one by one and align them with the master sound track. Each singer video can be aligned with the master sound track by aligning sound wave forms with the audio portion of the imported video. In some cases where a singer's video is out of tempo with the sound track, it may be necessary to snip it into segments and vary frame rates for each segment so that the sound waves will align to within a few frames. Once a singer's video is aligned, delete the audio portion of the imported video so that the only remaining sound is from the master sound track.
    (software used for this and following steps: Adobe Premiere Pro)
  2. Size, crop and mask singer videos so that they occupy the containers (or zones) according to the video concept. Then apply keyframed animation of these settings so that singer videos move with their containers.
  3. Apply filters such as contrast, brightness, color correction, etc. to singer videos so that they have comparable color values and skin tones. Lighting conditions and color temperatures of videos submitted by singers recording at home will probably vary. These should be corrected so that singers appear to have been recorded in the same space.
  4. Set keyframes and filters for singer videos and other visual assets so that they will move, scale, fade in, fade out, change from color to black and white, or vice versa at the proper times.

VIDEO OUTPUT

  1. Once all visual assets have been aligned with the sound track and filters have been applied, run a test export of the video to see if everything looks and sounds right. For a large project with many assets and filters, it may not be possible to preview everything simultaneously in real time. Given finite computer memory and speed, it may be necessary to work in the dark, editing and testing a few layers/tracks at a time. For a complex project, a video export may be the only way to see all of the layers/tracks on at the same time. Video exports from complex projects can take several hours, so it is good to run them overnight. When a video export is complete, view it and make note of places in the timecode that need to be fixed. After several renditions of this process, post a preliminary version of the video to a file sharing app (such as Dropbox) for the Music Director's approval.
  2. It is too late in the process for tempo changes, but if the Music Director asks for changes in dynamics or balance, they can be made to the sound track in the DAW app. Export a new audio mix (with no changes to tempo or duration) and then replace the master sound track in the video. After this, a final overnight render will generate a video suitable for posting on Youtube.