Saturday, 22 July 2017

2017 Uganda: Saga of the Drobo

UPDATE - It works!  After 3 hours of hard work, the drives came out at Room Temperature.
==================================================================

Here is the assembly video.

So on my last full day - Friday June 9th - we drove from Kampala to Fort Portal in the West end of Uganda.  This is the home of Mountains of the Moon University, the hosting institution for all the paper District archive work that Derek has been doing for several years.  The university is named after the nearby Rwenzori Mountains which are also called the Mountains of the Moon.  They top out at 16,762 ft. (5109 meters) which is the height of Africa's third tallest mountain.  They actually found glaciers up there - in Africa!

We were there to install a new DROBO 5c 5-drive enclosure to host all the digitized files of the archive work.  It will provide many TB of storage, allowing for future additions.

Unfortunately, once it was all set up and we were testing it, I discovered that the drives were extremely hot - so hot that I almost dropped the first one I popped out!  I measured the temperature when we got back to the USA and the case for each drive was about 145 degrees F.  You could actually have cooked an egg on it!

Disappointed, we brought it all back with us to figure out what was going on.  After much time on the phone with Drobo and HGST (drives) I was told that it was all operating "within parameters" and no one was going to help.  Drobo did say that if I wanted to ad a fan to the back of the enclosure that was fine with them.

So I did.  I had to make it removable and easy to attach once it all arrives back in Africa, as the fan can not be attached when the Drobo is back in its box.

Here are some shots of the process. Next step is to see if the USB fan I ordered will still provide enough cooling and be quieter.

The workbench view.  I cut some screen material to cover the unprotected side of the fan, which faces out.

The protective screen in place, with clearance for the fan blades.

One captive screw in place on the back of the Drobo unit.  These stick out to accept the fan and
still have enough thread showing to allow for securing the fan with with nuts & lock washers.

All 4 screws in place.

The fan and the Drobo ready to be reassembled.

Fan in place and ready for testing.





Saturday, 17 June 2017

2017 Uganda: UBC Diary

In June of 2017, I went to Uganda to consult with the UBC about their 3/4" U-Matic videotape library, and to discuss possible ways forward to digitize and monetize the collection.

I am also installing a DROBO drive array I brought for a district Records archiving project in the western part of Uganda.

Saturday June 3, 2017


My flight leaves at 6:13, so naturally I need to leave for the airport at 3:00 pm so as to not make my wife nervous.  An enormous Denali SUV pulls around to the end of our driveway at 10 minutes to 3, which is 10 minutes early.   It worked out since we had to stop at the ATM on North Campus so I would have enough USD to buy my Uganda VISA and still have some money left to exchange for local currency if needed to buy a SIM for Lester’s (very old) Chinese Samsung. Not sure if it will even work in Uganda.  It worked in Ghana, so I am hopeful.  Would not work in Athens, Greece last month.


First leg from Detroit to Amsterdam, about 7.5 hours.  Discovered this amazing secret room at DTW where you can go and check in and check bags and avoid the lines!  I was told that by the woman at the Sky Medallion front desk who said their system was down and that I should just go through that door - that I had never noticed before.  Apparently it is some kind of VIP check-in area.  I was the only one there, and there was only one woman behind the counter.  On my way out I asked if it was possible to use this place again, and she smiled and said “Well, keep it quiet and probably yes.”  Now that I know I’m going to try it every time!

I was in the front row of steerage, and so had to put everything overhead.  And my seat had a pull-out-and-up video screen that would not stay straight and kept slipping down.  But that was fine, as I tried to nap as much as possible.


2 hour and 20 minute layover at Schipol in Amsterdam, which was about the right amount of time.  I was able to clean up in the bathroom, find my gate, and have a yogurt and granola breakfast.  At least I felt I had something close to oatmeal. 

I really hope that the hotel breakfast includes oatmeal!

Second leg from Amsterdam to Kigali, Rwanda was also about 7 hours.  Sat in 15A next to a Bioengineering undergraduate major from Texas A&M who was on her way with a group of fellow Aggies to Rwanda to spend the first month taking classes (mostly language classes) and then some others about the bio-med machines they were there to install and/or repair, and then train the Rwandans to use.  


All of us that are continuing on to Entebbe airport in Uganda stay on the plane for the 1 hour layover here.  Which is when I am now starting this diary.  I said goodbye to my young Aggie friend with the admonition to “Go save lives and change the world!” 

(I need to get this autocorrect under control.  It had that poor girl going out to save olives!)

Last leg was a short one.  Stayed in the same seat, and this time my companion was a woman from Rwanda named Connie who was on her way to Toronto for her youngest son’s High School graduation (the flight to Amsterdam starts in Kigali).  They had lived in Toronto for some time and both her son and older daughter stayed there.  Barely enough time to distribute cookies and we were landing.

Took FOREVER to get my SIM card at the airport.  I also changed $60 and got back $250,000 Uganda shillings (UGX).  An exchange rate that far apart and it feels like you have Monopoly money.  

I was literally the last person to leave the airport from my flight, and by the time I finally got outside the person that had come to meet me had almost given up.  They were thinking maybe I had missed the flight, since everyone else had come and gone.  I was glad they had waited.  The Hotel pays them for the ride service and they did not want to have made the trip and not get paid.


My room.  Tight quarters, but nice.
Entrance to the Speke Hotel.  Only black people had to go through the metal detector.
(The entry is flat; it is distorted due to stitching)
Hope to be able to upgrade the Samsung phone to be able to call the US tomorrow, but for now I called Jeri using WhatsApp on wifi.  The Samsung will be quite useful locally in any case.  Unpacking and off to bed now at 12:35 AM on Monday morning.  I left on Saturday afternoon.

Down to breakfast by 8 am, then meeting Derek here at 9 AM, and then off to the UBC.  

They had oatmeal at breakfast!


Monday June 5, UBC

The sign at the UBC building.

The interior atrium of the UBC building.

Arrived and spent the morning in the company of Malachi & Jacob, both UBC employees.  They have the impossible task of trying to organize mountains of material - literally boxes piled on boxes, piled on shelves, filling every corner of every room that is in some way dedicated to the library.


A 360 image of Jacob's office.  It makes it look like it has more room than it really does.
This is the land of Idi Amin, one of the most ruthless dictators to ever come to power.  
They are correct that there is likely some value to be had by the State from this archive. 

This is bureaucracy at its best - or perhaps its worst.   We visit room after room of badly stored U-Matic and other video tapes, the 'other' mostly being Betacam and various formats of DV.  They had been burning some things to DVD discs, as there is no in-house means of taking digital files and storing them on a central server.  All of these assets are in an extremely vulnerable position.

Red dust accumulation all over the tapes & boxes.

Almost all the tapes have been stored horizontally, which is not good.

One of the librarians and her record collection.


And the "best of the worst" was this:  piles of films, rotting in various forms of rusting decay in metal cans stored - literally -in a disused lavatory in an unused building.  

All that was missing was a sign on the door saying “Beware of the Leper”.   
 (Thanks to Douglas Adams, author of "The Hitchhiker's Guide to the Galaxy")  

Seriously: this is an unused bathroom.

The yellow film can near the top labelled "Series Scooby-Doo" is the cartoon.
They have no hardware for playing anything but 1/4” audio tape and VHS videotape.  I have not seen the alleged working VCR, but have seen 3 Studer A807 real to reel decks, only one of which is said to be in working order.  True or not, the tape path, and the deck in general, are so dirty it would take considerable effort cleaning them to get them ready to test.


One of the non-working Studer decks.

This AKAI was in one of the tape storage rooms.  The windows had been left open for years.

They realize the gear is in bad shape, but there is no money for tools, supplies, or maintenance for anything but the equipment currently being used for their broadcasts.  

As they said in Casablanca: 
"I am shocked - shocked! - to find out that there is gambling going on in this establishment."

Over lunch in the canteen, we discussed a possible new way to approach the powers that be with a plan that might allow the UBC to keep a constant chain of custody of the tapes.  It seems that this has always been the sticking point: whoever would do this for free (French and German TV) want to keep copies of the digitized material for their use.  The UBC does not want that to happen.

The "Lunch Ladies" at the UBC commissary.

This has become the great struggle between Africa and the North: ”They come and take our heritage and leave us nothing!”.  It is an understandable position after so many decades of actually being screwed.

So my idea is to frame it differently.  They do seem to understand that the best/only way to get this stuff digitized is to put it in the hands of professionals that are capable of keeping the obsolete technology in best working order to get one optimal playback from each of the tapes that the UBC wants to digitize.

The fear is loss of control over what happens if it leaves their custody, and it is not an unjust concern.

My idea is that instead of shipping tapes at great expense to somewhere out of their control to an agency with an interest in using the material, to instead send it in batches to a service bureau in the luggage of a UBC employee who will oversee the process and return with the analog materials and the digital files of those same tapes.

This does 2 things: 1) it maintains the chain of custody in the hands of the UBC and 2) it puts the work of digitizing into hands of a business whose continued existence pends on the trust of their clients.  They have no inherent interest in the material, only in getting a good recommendation from the client as to their integrity and quality of work.

Tuesday June 6, UBC


Derek & I had Indian food at the hotel restaurant for dinner.   I'm sure other things happened, but I did not make note of them.

Wednesday June 7, UBC


Went to see the Engineering crew and the Presidential Press Unit (PPU) people. The PPU is about to get funding for their own version of the same kind of archive project, but can't talk to our guys or share any ideas.  And of course sharing resources is out of the question.  What a world.

Turns out the Maintenance Engineers have at least one fully functional Studer audio tape deck.  This is good, because neither of the ones in the Library are working.

I then talked to the real IT guy in charge and, once I found the right questions to ask, got good information from him.  

Apparently there are 2 independent networks here:  the Corporate network and the Media network.  And there is no connection allowed between them.  This is for security reasons and sort of makes sense.

No problem.  The chief IT guy said that as soon as the cable he needed had been requisitioned and delivered he could easily run the wire from the Media server room to the room where the Library folks would be working on the archive - 4 or 5 months tops.  Just to get the wire.

Derek & I met with the Asst. Managing Director who was very receptive to our plan moving forward.  We meet with the Managing Director tomorrow.  His wife just delivered a baby boy and he has been out a lot.

Derek began sorting tapes with Raymond, the first intern, in the Room of Doomed UMatics.  I worked with Jacob on the Database.  We added some fields and renamed others and began building the controlled vocabulary.

Thursday June 8, UBC


Had 3 interns that really dug in to the stack of red-dusty UMatic tapes.  


The result of having open windows after the AC units failed.
The earth around Kampala is very red.


Just one of the many dusty and dark rows of shelving.

I had them move a bunch of boxes and piles of tapes that were in the way all over the floor of that room out into the hallway/common area.  People were amazed that some work was actually being done around the archives.  The interns labeled all the shelves in there with numbers according to the plan I laid out and began the task of sorting the tapes between those from an external source like the BBC and those with original UBC content.  The lighting was so bad in some areas that they had to wear my headlamp.

Labeling shelves so tapes can be located.
Between the dust masks and the headlamp, it looks like they are working in a coal mine!

As Derek was at a meeting that went longer than anticipated, I met with the Managing Director accompanied only by Malachi,.  I laid out our understanding of their concerns and our idea for a new approach and how it seemed to dovetail into their existing facilities and capabilities.  I expressed that the UM would be happy to partner with the UBC to seek funding from the NEH or similar organization.  

I emphasized that any funding organization would insist on allowing for research access and discussed several ways in which the material could be made available securely, such as only allowing outside access only to the catalog, to only allowing internet access to degraded or watermarked versions of the actual video.

Derek showed up as we were wrapping up.  The MD was very happy and Derek reiterated the points that 1) the catalog work needed to proceed with all haste and that a laptop was needed to continue, and 2) we would send a short summary of our mutual understanding about moving forward quite soon and then follow up with a longer more detailed plan of what we  were proposing.

Both UBC employees and their interns thought this experience would help them moving forward in their careers.

They are being given a unique opportunity here: they are in at the very beginning of organizing and cataloging an archive of legacy material.  This will be a once in a lifetime opportunity for most of them, or a rare opportunity at the very least.

Around 4:30 pm, when it was clear that we were  not going to see Jacob again today and so could do no more work on the catalog, I left with Allan, an intern, as my tour guide,  We stopped at an ATM so I could pay for our boda-boda rides and the entry fee for the Kaddaffi mosque, our destination.    

          Here is a 360 video of my ride.


The front of the mosque.


About half the height of the Washington Monument and only a stairway to the top.  It was a long, but fun, climb.  I never looked up because I never wanted to know how much farther we had to go.  


Looking down the minaret's stairway.

Just a nice image of window & shadow.

The view from the top was spectacular!  


View from the top of the minaret.
That night we ended up eating pizza at the hotel for dinner.  I don’t feel too bad, as it was a treat for Allan and we had been eating at the UBC commissary every day for lunch. That is real Ugandan food.

Allan was a wonderful tour guide and we had great conversation over dinner.  I told him to make the most of this internship opportunity and also told him about UMAPS.  We’ll see if he ends up applying.  He want's become an academic and a teacher.

Leaving at 6:00 am tomorrow for Fort Portal, in the western part of Uganda, where Derek has been doing his paper district paper archives work.  I will help them get the DROBO we brought set up and working and get a chance to see more of Uganda before I leave on Saturday.

Januario Mubangire, our driver, arrives at 6 AM.

Friday June 9, Fort Portal 


Arrived at Mountains of the Moon University.


The Mountains of the Moon are nearby and separate Uganda form the Congo.

Derek has been working with colleagues there to scan and archive collections of regional government documents.
The room where the documents are scanned (scanner is covered).

The archive room , both paper & digital.

Our MMU colleagues and the DROBO to be installed on the computer in the background.
Spent the afternoon setting up the DROBO 5C on the brand new HP computer the has been purchased for the archive.  Things went well, but took longer than expected due to the slow internet connection.  The first thing the 5C does is connect to DROBO and update the firmware.

However…. At that point, while the Dashboard “Capacity” tab says I have the expected amount of about 14.5TB of space, when I go to Format the drive under the “Tools” tab it will only let me format it at 64TB.  Same thing happens if I try to format it under the Windows format option.  I even attached it to my Apple MacBook and found the same issue of Dashboard Capacity being correct, but only being able to format at 64TB.

But the strangest thing is that when I popped out one of the drives to see if it would rethink what was going on, it was so hot I almost dropped it!   Thus begins....

The Saga of the DROBO


So I waited until 7:00 PM Uganda time so it would be  8:00 AM Los Angeles time and called Drobo support.  On hold for 20 minutes but then hung up, worried about running out of minutes on my phone.  Called them again after dinner and spoke with a PERSON (!) in sales, explained my plight, gave them my case number and asked them to put it in front of a tech support person.  They promised, but I never got even an email, let alone a phone call.

Tried once again at 2:30 AM Uganda time after being woken up by an extraordinary chorus of dogs in the night.

This was an extraordinary canine chorus.  It stared with one barking, then a few more, then about a dozen in a riot of barking at something.  Then silence for a minute.  But then they started up again and went into more of a wailing chorus, actually harmonizing and sliding up and down the scale for over a minute.  I was just getting my phone out to record it when they stopped.  It was amazing.  So that’s when I called Drobo again, on hold for 30 minutes, then went back to sleep.

Back in the US


I found out, once back in the US with access to an instant-read thermometer, that I could have literally fried an egg on those drives: they were running at 145˚ F.  An egg fries at 144˚ F.

I called tech support for HGST, the drive manufacturer, and the guy who answered was also shocked at my reported temperature.  He put me on hold and went back to talk to the real tech guys.  When he returned he sheepishly told me "That's within spec" and so they would be doing nothing about it.

And when I finally got someone on the phone at DROBO, they too said nothing was wrong: the unit was operating within specifications.  I agreed that this was technically true, but pointedly asked why they would design a drive enclosure whose normal operation kept the drives inside hot enough to fry an egg. They had no comment.

Two days after I cobbled together my solution and sent the DROBO back to Africa, DROBO emailed and send that my RMA exchange had been approved and to expect a replacement in a few days.  Sheesh.....

OK, Back in Uganda now


(see my blog about how I fixed the DROBO here.)

Saturday June 10: Fort Portal, and then back home


After waiting on hold with Drobo for 1 hour 50 minutes this morning, I ran out of time on my Samsung phone.   Derek & I decided to bring the whole DROBO unit back and return it for something that has better cooling.  I am very disappointed in the whole thing, having had such good luck wth DROBO in the past, to discover that it was behaving so baldy and being their newest unit.

While waiting for Derek to return with the DROBO boxes, I saw a Sunbird and a Bulbul in the trees in from of my room.  The Sunbird is a tiny bird, larger than a hummingbird, but has a long dipping curved bill and goes around to the long orange blossoms in the tree outside, dipping in for nectar.

The Bulbul is more thrush sized, blue, and was just passing through.  

The Bulbul is the blue spot, middle right side.


Derek returns with the Drobo boxes, and we return to Kampala.


A typical termite mound.  They are everywhere.

Typical road side view as you go from country to a village. 
There are always vendors along the road side.

This video clip shows another road side view, including lots of the vans that make up the mass transit system.  Most people depend on these vans, which hold up to 11 passengers, to get to and from work everyday.  Some rides take hours.

                               

Just stopped in Wribare to fix a flat tire.  It was quite the adventure.  


Changing the tire.

Januario knew where the jack and spare were located but, this being new vehicle for him, had never had to use them.  Turns out the spare is lowered from beneath the carriage under the rear of the van.  


There is a square bolt top visible in the door seam and that is what cranks down the bar securing the frame holding the spare.  That bolt has to be turned - quite a lot, as it turned out - in order to get the clearance needed to separate the pieces holding up the spare. 

They did not figure out that the handle for cranking up the jack doubled as the wrench for cranking the bolt, so Juanario found a pair of pliers and kept turning the bolt 1/4 turn at a time for a bit, then having a couple of strong guys that had gathered (this was, after all, a big event) pull up on the horizontal retaining frame while he tried to get the vertical piece disconnected from it.  This went back and forth, and then eventually they were banging on the top of the bolt as well as lifting.  I kept thinking that they really need to keep turning that bolt, and even said so once, but no one responded so I kept quiet.

Then Abdulah, the former Chief Engineer for the district, came over and said the same thing and proceeded to do just that with the pliers, and finally it all came free and they began the work of changing the tire.

But it just seemed to me that this was a really stupid system if it was not self-contained.  It should not require that you have a pair of pliers on hand.

So while they were changing the tire, I took an extra piece of the parts that one would assembled to create the crank for raising the lift and tried to fit it onto that square bolt that they had been turning with the pliers. 

I tried, but it did not quite fit.  I felt the edges of the square bolt head, and found that all that banging had flattened them out so the head was wider than it was supposed to be.  This made it impossible to fit the "wrench" onto it, the way it had been designed to do.

Taking out my trusty Leatherman, I filed down the edges until they were smooth, and voila! - it fit.  So putting the flat tire back in place of the spare only took about 30 seconds.  And now Januario knows how the system works.  

A quick side-of-the-road bathroom stop, and on the road again heading back to Kampala.


Some serious African sun...
The trip to the airport was amazing. There is a new highway not yet open - but soon to be - between Kampala and Entebbe, where the airport is located.  But if you know the way, and pay a small sum to the people 'guarding' the secret entry ways, you can get on it and use most of it.  We were on and off various parts of it al the way to the airport...



That's it for this trip.  Check out my 2018 Uganda entry.

-t

Friday, 3 March 2017

Sunday, 5 June 2016

1988, in a former life

So one of the things I used to do a lot was record and/or do live broadcast mixes at music events.  I still love dong it but rarely get the opportunity any more.

First, here's what I looked like when this happened; nice, eh?

    


The letter below (not an email!  It is actually signed!) was sent to me after a particularly fascinating incident at the Montreux Detroit Jazz Festival in 1988.

The author of the letter and the guy in charge of the broadcast on WEMU stopped in where we were mixing the music from the Pyramid stage at Hart Plaza.  After a few pleasantries he asked "So are you using a lot of EQ or something in your mix?"  I said "No, just listen to what is coming out of the speakers.  Why do you ask?"

He said he had been listening to the broadcast on the monaural radio in the TV in his hotel room, and when the broadcast switched from the other stage to ours the sound was awful!  He came scooting right over to our mixing room and started the conversation.

So once he heard what we were actually doing he said it really did sound great; superb, in fact.  After a little bit more genial conversation I had eliminated any doubts from Jim's mind that it was me creating the problem.  I knew that what we were doing was fine: I just wanted to get to the bottom of the issue.

So, being an engineer kind of guy, I started looking for what I thought was the likely culprit: one of the Left/Right lines from our room to the booth had been wired out of phase, effectively canceling most of the audio in our mix and leaving sounding very thin and awful - but ONLY when heard in mono, as it was on his hotel TV.

I followed the approximately 700' of wire all the way to the final broadcast feed mixer, where I discovered that indeed, at the last connector in the line going into the back of that mixer, the Right channel balanced line had been incorrectly wired and was therefore electrically canceling out most of our mix.  And it had been that way for a day and a half.  Apparently the Engineer in charge of the broadcast had not bothered to check the lines.

I gathered up a soldering iron, pliers and a tabletop clamp and quickly fixed the problem.  Ten days later I got this letter in the mail.  Ain't life grand?



Thursday, 19 May 2016

2015 WCBN Tapes at the Bentley Historical Library

In the summer of 2015 I spent some time at the Bentley cataloguing the tape archive of WCBN.  These tapes had been there for several years waiting for this to happen.  The next step is to find money to hire a student to digitize them - hopefully this summer.

Here is the first photo of that work: a bunch of reels in a box.  I'll add more photos and some video soon.




Tuesday, 14 April 2015

2015 Ann Arbor: Fool Moon

Fool Moon was way better this year than last year: it didn't snow or have 50 mile an hour winds.

On Friday April 10 Vodo Jumato - VO-ratima Orawannukul, DO-nald Harrison, JU-stin Dykhouse, MA-rtin Thoburn and TO-m Bray - set up their giant projection monolith and painted the side of the office building at Main & Washington with appropriately foolish visuals.

I was prepared for inclement weather this year. I had tarps and a plan.   Once I had a couple of people to help it only took about 5-10 minutes to completely secure it from the wind and rain.  And once again we had Donald's guitar amp strapped to the lower plank for ballast.

Here are shots of "before" and "after" wrapping:

           

In the above photos you can see the planks we used as a work surface and to support projectors.
We had a small projector pointing out the back showing the twitter feed.

In the above photos you can see the 'windows' I made out of 8 mil clear plastic, for the projector up top and for the VJs below.


The view from the top, showing the VJ work surface, Voratima and Donald.

Martin and Donald at work.  Martin's laptop was a source for Donald's laptop
via a Blackmagic Intensity Pro.  You can see the window.

Donald's laptop, my darkroom timer and the Hard Hat Cam.  We used the
timer to turn off a light to remind us it was time to run a sponsor logo.

The view from outside the window.

Me! From outside the "Rear Window" from where we projected the
Twitter feed on the wall behind us.

My Live Feed Hard Hat Camera:

Tiny camera on the left of this photo, batteries and Xmtr on top.

Battery packs for the camera secured with the camera headband.

Another side view.

I walked out into the crowd and got in front of someone and said "Look at the wall!".  When they did they would see themselves as part of the projection.  It was a big hit, even though it was not great video quality.  The piece missing form the QTVR is the headlamp I attached so I could both aim the camera and light my subject.

Here is a link to a QTVR movie of my live video camera hard hat

You can download it.  It will only be QTVR if you use Quicktime 7. QT 10 will open it as a still.

-t