VLOG: The BIG Data Storage Problem
February 5, 2016 by warzan
For some website features, you will need a FREE account and for some others, you will need to join the Cult of Games.
Or if you have already joined the Cult of Games Log in now
What difference will having a FREE account make?
Setting up a Free account with OnTableTop unlocks a load of additional features and content (see below). You can then get involved with our Tabletop Gaming community, we are very helpful and keen to hear what you have to say. So Join Us Now!
Free Account Includes
- Creating your own project blogs.
- Rating and reviewing games using our innovative system.
- Commenting and ability to upvote.
- Posting in the forums.
- Unlocking of Achivments and collectin hobby xp
- Ability to add places like clubs and stores to our gaming database.
- Follow games, recommend games, use wishlist and mark what games you own.
- You will be able to add friends to your account.
What's the Cult of Games?
Once you have made a free account you can support the community by joing the Cult of Games. Joining the Cult allows you to use even more parts of the site and access to extra content. Check out some of the extra features below.
Cult of Games Membership Includes
- Reduced ads, for a better browsing experience (feature can be turned on or off in your profile).
- Access to The Cult of Games XLBS Sunday Show.
- Extra hobby videos about painting, terrain building etc.
- Exclusive interviews with the best game designers etc.
- Behind the scenes studio VLogs.
- Access to our live stream archives.
- Early access to our event tickets.
- Access to the CoG Greenroom.
- Access to the CoG Chamber of Commerce.
- Access the CoG Bazarr Trading Forum.
- Create and Edit Records for Games, Companies and Professionals.
wooohoo keep it up boys
All the best with that guys, wish I had the expertise to help you.
@warzan For your IDE drives, you can purchase IDE/SATA to USB adaptors which you can use to access those old IDE drives. Locally they sell for $20-$30 Canadain. Most good computer supply stores should carry them. I know USB is not an ideal solution for permanent mounting, but if you are physically archiving the data and then accessing it infrequently, it might be the best bang for buck option.
http://www.bestbuy.com/site/thermaltake-blacx-hard-drive-enclosure-docking-station-black/9062055.p?id=1219699418029&skuId=9062055
USB IDE Dock…lets you hot swap drives to transfer data. A slight improvement over enclosures, you don’t need to assemble anything around it. Also reads smaller laptop drives.
whem u crack open those external HDs open guys, expect you obsolete pile to increase, because some have none standard HDs inside them 🙁
I need an edit button. sorry for the bad sentence…
You should be able to use Adobe Bridge to search a file or volume by tag. If that’s the case you just find some poor volunteer to pick through your archives to standardize the metadata in Bridge as you duplicate your drives.
For off-site storage, is it worth looking at some kind of cloud storage?
I don’t think so for their needs, a gig is fine and free but after that, your looking at at a monthly cost which adds up when it is per gig. Also most companies operate out of the US, the cost of uploading is phenomenal.
May be I am wrong.
You are not wrong. Personal levels of cloud storage are free, with costs incurred after certain levels, but business levels of storage are beyond what BoW could reasonably afford.
I am hybridising on-prem to IaaS just now, costs are mental.
DO NOT OPEN THE HDDS. You’ll kill them instantly. * Unless you mean just taking the drive out of the consumer case. But taking platters out of the HDD will turn them into drink coasters. * If you don’t need to retrieve data very often I would use Amazon’s AWS Glacier service. Also, if you need local storage I would start using solid state laptop drives. They’re a bit pricier but the size is smaller, there’s no moving parts, they’re not magnetic storage. Honestly though, I would use hosted services like AWS, or Rackspace. Or even stuff like Dropbox. A… Read more »
Cloud storage is probably the way to go these days not sure i would risk Dropbox they have a few security issues. I am guessing that it’s not just the image and video stuff that needs to be approriatly stored but all the business support data aswell.
Dropbox has definitely had security issues in the past, I think they’ve corrected that. Their business + enterprise offering is definitely better than the free offering. But, I’m probably biased as I’ve got friends employed there as developers. But, there’s lots of alternatives to Dropbox, I only mention it because it’s well known and super easy to use. Google offers similar services for similar pricing but with significantly more storage (order of magnitude more). You could also just something like a Drobo in the office for working files and then AWS Glacier for archives. Or build your own completely hosted… Read more »
Wouhouuuu, keep it up boys.
I am a professional Information manager with nearly 20 years experience if thats any help @warzan i dropped you an email.
This may seem like an amateur question but once a video has been released, why do you need to keep all of the production footage? Is there a legal mandate for you to maintain records of all production footage or could you get away with just keeping the videos themselves? It may be a tough decision but unless you really need to keep the production footage, you could significantly reduce your data footprint by just backing up copies of the videos as they are when they get released.
My company uses Amazon but there are other options.
https://aws.amazon.com/
In particular check out the archive storage options as those tend to be cheaper. I think we use glacier for long term data storage.
https://aws.amazon.com/glacier/
Why don’t you burn every video on a cd-rom?
Hi guys, From someone who works in the industry, can I suggest Amazon Web Services (AWS)? They are a massive industry leader for cloud computing and storage. Even giants like Netflix use them. They offer several different cloud storage methods – but for archival in particular they offer something called Amazon Glacier. It’s specifically designed for data you’ll rarely need to access, but you’re talking incredibly cheap prices for storage – approx $0.01 (1 cent) a month per GB. As it will all be hosted in the cloud, you’d use AWS’s own console / website to access everything so this… Read more »
So taking a picture of each page on your phone wouldn’t help?
Taking photos just creates more data so you’re not solving anything. You need to get someone to hand sketch a copy of every page onto paper and the compile them into books.
Machine spirits is the way forward…
New BoW Archive Solution:
http://vignette2.wikia.nocookie.net/warhammer40k/images/5/5c/Scribe.jpg/revision/latest?cb=20120712015048
I had the problem some time ago of accessing an old IDE drive. Could i suggest one of these:
http://www.amazon.co.uk/AKORD%C2%AE-Drive-Reader-Convertor-Adapter/dp/B00EQ7UEK4/ref=sr_1_2
Here are the storage boxes you are talking about @warzan http://www.aquaphoenix.com/application/sales/index.cgi?item=18 4 boxes are going to hold your 80 discs when you crack open the ones from the case. Many of the newer ones will be SATA inside anyway. (Maybe some of the cases and internal gubbins will go towards the terrain bitz box 🙂 The problem of the technology moving on is already at the door SSD drives are getting cheaper and with 100 times the access speed they are doing away with the bottleneck that has exsisted in computng for a long time. I know it’s time consuming… Read more »
My work used to throw a bunch of those exact same boxes/foam away every day. I’ll check tomorrow if that’s still the case. They recently got a lot more “green” and implemented processes to eliminate waste, but if they’re destined for a recycling plant somewhere I could probably lay my hands on a few of these.
Don’t envy you guys on this one. Having worked in IT contracted to several motor manufacturing groups, I can tell you storage and archiving has never been an easy task, especially given how much data you’re trying to store. I’m guessing a ballpark figure of 100Tb roughly? With cloud storage, you’re then looking at several thousand pounds a year at the very least, pretty much the same as a JBOD array. So alternatively we’re back to storing the hard drives outside of the casing but that can be just as much of a pain. The polystyrene boxes are fine for… Read more »
well you asked for it. Woo Hoo Good luck guys your awesome. Keep it up Boys
Tough task to do on a budget. Woohoo! Keep it up boys!
Thanks for the peeps behind the curtain. Woohoo keep it up boys 🙂
wooohoo keep it up boys
You have another problem you did not think of. Magnetic medium only have a very limited shelf life. We are having issues getting the 7 year requirement for PCI data at our company, magnetic tapes have a shelf life of 2-5 years. The suggestion is that the data get refreshed, rewrote to the medium every 2 years. So you will also have to consider that too in your storage of data. Blueray/DVD written disks only have a very limited shelf life too.
The Beast can NOT be contained, it just keeps growing and growing.