Asking mostly because I have fuckloads of video courses, plus a number of movies, that I have yet to even check if the content is as good as their titles imply and I really feel like I’m mostly hoarding this stuff because I have no fucking clue.
How do you avoid “hoarding”?
I dont. Hard drives are increasingly cheap/large. I have to really dislike something to delete it. I have a fair amount of content that I don’t really plan on watching again, but someone I know might like it so i just leave it typically.
I have to really dislike something to delete it.
The velma tv show was the last item I just deleted.
But for me this is the same story. I’m up to 400TB… I’m just over half full. I’ve got plenty to go, and if I make to to 75-80% full, then I’m going to get me a 45 or 60 bay server and upgrade from my 36 bay one. 6 of the bays are wasted on SSD caching currently… Just finding a chassis that doesn’t waste the 3.5 inch bays on 2.5 drives would allow me to add a full vdev(another 100TB…).
Old chassis can be had on ebay relatively cheaply.
Do you have an offsite backup? Or do you only backup specifics? Like 10-20% of that?
I do not have full proper offsites… yet.
I run proxmox, so if it’s live on a server it’s probably on my ~70TB (really 40*2TB ssd) ceph cluster. Which makes 3 copies across the 5 boxes, so it’s more like 23TB of usable space for all my vms and such. The 400TB of storage is Truenas is really closer to 300TB after all the losses in raidz vdev and hot spares and what have you, there’s 30x 16TB SAS seagates in the box, of which 2 are hot spares and 7 are parity for raidz1… For things that are slow or linear loads (a movie file could be a good example of that type of workload!). Backups of the the proxmox boxes… and mass stored stuff, 99% of it I could easily obtain again if I had to. Although I’d probably be pretty flustered about it.
Truly important stuff gets written to 100GB bluray(s) (specifically m-disc blurays) and put in the safe. I do this probably about once a year or so…
My dad was in the process of setting up his own cluster that’s running 14TB drives rather than my 16TB… When he’s finally done I intend to requisition probably about half of his space for offsite storage (maybe more). I’m figuring about 100TB of space is what I’ll have there. Maybe more. He’s about 65 miles away from me, different electrical grid and all.
So the count as it stands now. Everything running has at least 2 copies on 2 mediums (ceph cluster, and spinning rust). My “linux iso” repositories only live on the spinning rust storage, but is low priority anyway. Super important highly sensitive shit lives on at least 3 copies and 3 mediums, although one of the mediums may be out of date and none is offsite… Though it’s rare I add to this category. There is plans for adding another copy of data, offsite on harddrive storage for most of my dataset as it is now.
Truenas usages:
And here’s Ceph
Your setup: goals.
Where are you sourcing your hardware? Decommissioned enterprise DC stuff or are there options outside of the enterprise space that enable this type of setup?
Ebay and decommission. I got really lucky on my SSDs, those were all from a decommission. Company was going to pay an ITAD for destruction. I picked it all up and wiped it on site. The rest are relatively cheap hardware, supermicros and such… but with enough of them you can build a resilient cluster.
A lot of my stuff is Ebay… I did recently purchase a new rack as probably the only “new” item I have in regards to my setup. The old one had issues… and I didn’t want to deal with thrifting broken racks anymore. And I needed a taller 45U rack rather than a 42U standard rack… Also the more depth means I can accommodate the 60 bay server in the future if it comes to that.
But things like 40gbps networking… ebay. The proxmox servers are decomissioned. the truenas server was ebay. switches was ebay… Oh! The firewalls… That was new purchase. I am stupid lucky to live somewhere with 8gbps fiber. I needed real horsepower to push that with IDS/IPS enabled. So this was a new purchase from supermicro. The SAS spinning rust drives I picked up on Reddit homelabsales or something like that a while back. PDU’s were ebay… UPS were ebay… Expansion batteries were craigslist. Most cables were new from FS
Previous versions of my rack were government liquidation/auctions. My dad has a lot of that equipment now. I found one auction that was 1400$ that was basically a whole racks worth of shit… most of it pretty usable 12 and 13th gen dells. And another auction for 600$ that had a dell m1000e with some 4TB of DDR4 ram…
But you can do a lot of this shit with a cluster of little N100 boxes if you really wanted. I just happened to get my hands on enterprise level equipment… So I joined the Romans…
These are my thoughts exactly; piracy is preservation.
who avoids hoarding? we are saving it for the future.
Exactly! I assume that my little 4tb external drive full of movies will one day be the only usable relic discovered of our civilization, so I must plan accordingly lol
I have two servers, a >100TB rack-mounted Supermicro archive that doesn’t get fired up often, and an Intel NUC that runs 24/7 but only draws 5W at idle. The NUC with its mere 4TB SSD is only for content I’m actively watching which gets deleted immediately afterwards. Running just the Supermicro made more sense when I had a terrible internet connection and had to wait for everything but I moved to an area with 1Gb+ connectivity a few years ago and subsequently needed to save on energy costs.
I feel like the real question you want to ask yourself is, “how likely is it that this particular content will still be available on Usenet/torrents in a few years?” Some stuff is much more niche and rare while other movies/shows each have over a dozen redundant releases, at least a few of which will more or less always be available somewhere. To put things in perspective, it also helps to do an analysis of how much you’re spending each month in order to avoid what you would be paying in streaming and licensing costs, including hardware, power, and connectivity. If that ratio gets too high then it’s time to scale back.
I archive, never delete or stop seeding. I would just delete when you need space if you don’t want to buy drives.
I do not avoid hoarding.
I’m like a dragon with a media treasure stored in high capacity industrial HDDs.
Someday the age of pirates may come to an end, and I want to be prepared.
Same. I’m a Doctor Who fan. I don’t need to learn lessons twice. My grandchildren will be able to watch the dumb shit I grew up with one way or another!
I usually just pirate ebooks, rarely videos or games, so my piracy doesnt take up much space. For videos I just keep em around since I have the space, and if i need to clear space I can just delete something I’ve already watched. And tbh I don’t remember the last time I pirated a game, I don’t play games a lot and the ones I do play I own legally lol
Oh I do pirate music too. It’s currently not taking up much disk space so I’m not bothered. I would probably just buy another drive if i ran out if music space.
Like others here, I tend to only download when I find (or remember) something I like. Most of the stuff I have was either downloaded to watch or listen to right away. Others are things I watched at some point, or that I just finished and really liked. Especially anything exclusive from the current services since they don’t bother to release physical copies (or even legal digital purchases for that matter). When they do release a disc they fuck up getting any money from me by virtue of a HD/4k being only released on DVD.
After seeing the more and more open statements and updated TOS’s about losing things if they just decide to ditch an outlet. I finally got around to getting a BDXL drive for my PC and flashed the unlock firmware. So I plan to rip my discs to have all the access I can give myself. Sadly I really really need to commit to getting some actual capacity drives, and move my server to a dedicated PC and not just keep running off my daily PC (though it can handle double duty pretty easy after a couple of years of big upgrades).
Weirdly enough my legal digital libraries tend to have more of an issue with “hoarding” if there are like “Steam sales” on whatever service. Also tend to get things that are part of Movies Anywhere since it is basically the closest thing to having a bit of protection of not losing stuff if any one service closes. Helpful for my current lacking of proper drive space. And I plan to rip those streams once the other PC gets built (or until I build a new main PC and setup the current as dedicated).
How do I avoid Hoarding? Well I have a total of 2.75tb of space, so when it gets a bit full I go through and delete shit we watched already so I have space for more stuff
I have a lot of ebooks that I download for university research, hobby learning and friends who ask for help sourcing books. I put everything in my calibre library, which is great for metadata management (tip: I have it set so new books that I’ve just imported get a tag of “new”, which I remove when I have processed their metadata. This allows me to chip away at ensuring the metadata is correct and good, even if I don’t do it at time of import).
Anyway, at one point I found myself at risk of becoming overwhelmed by books, because if I’m wanting to learn some category theory, for example, I’d have multiple books that seem to be relevant. Some of them were recommended by programmers, some of them assume a higher level of maths background knowledge, some of them are more fun to read — once upon a time I might’ve known which was which, but if there’s a significant gap between me downloading stuff and using it (which is often the case, I’m quite opportunistic with book recommendations), I may forget. Making a note of why I downloaded a particular book is something I’ve been trying to do more, so I can identify the useful things at the right time — the calibre notes field can work for that, but I’m still figuring out how to manage this in a wider sense because I do a lot of reading and it’s easy to forget why I’m reading a particular thing. I think I have a calibre plugin to show which things I’ve read also.
Another related thing is that I will take a cursory look over a book when I download it, and I may delete it and not put it into my calibre library. This feels significant because downloading a book doesn’t make it one of my books, ‘taking it home’ and putting it away on my ‘bookshelf’ makes it mine. In short, I try to be mindful in my curation activities, recognising that doing it in big clumps with my whole collection doesn’t really work and that pruning little and often helps more.
hey you mentioned category theory, any math community on the fediverse? r/math is sometimes gold but most of the time cringe, because… reddit
Sorry to reply to this so late, I procrastinated because unfortunately my answer is that I don’t know of any communities, perhaps because I’m a scientist who loves maths rather than a mathematician.
However, I will use this opportunity to share some fun stuff from people I like.
https://youtu.be/H0Ek86IH-3Y by Oliver Lugg on Youtube is great. His channel is very eclectic though, and there isn’t much pure maths. I love his shitposting tone though, and he has a discord community that were pretty mathsy when I was in it.
A blog-type site that I enjoy is Tai-Danae Bradley’s https://www.math3ma.com/about, largely because I’ve discovered many other cool researchers through her site.
I also really enjoy Eugenia Cheng’s books, especially as someone who is interested in understanding how to write good scientific communication that is accessible without “dumbing things down”. I recently finished “The Joy of Abstraction”.
Apologies that this isn’t what you actually were looking for. I share your distaste at Reddit: I have used Reddit occasionally for those niche communities that aren’t available elsewhere (yet!), but the atmosphere is increasingly toxic. I fear that smaller communities that flee are congealing in harder to discover places, like Discord.
thank you
Sorry to reply to this so late, I procrastinated because unfortunately my answer is that I don’t know
relatable lool, “I can’t do it rn cuz I still don’t know how to perfectly”
I only dl shit i’m interested in. After watching if it’s gonna be rewatched, it goes on the fserve. If not, delete
I’m concerned that crackdowns on pirating will come sooner or later. At some point it may become too much of a hassle. So I’m hoarding a lifetime of old movies and games to hold me over.
Just keep seeding the torrents. As long as the torrents for stuff is healthy, how could it be cracked down?
Enough countries ban torrenting so you cannot even uses vpn to get around it.
How do they do that?
Ask Germany. Torrenting without VPN is almost imposible, you’ll get sued by copyright lawyers.
Exactly, so you can use a VPN to get around that.
You use a vpn to other country without that ban. Once enough countries apply that ban. Where are you going to VPN to?
Why would any country ban torrenting? Torrenting isn’t illegal, downloading and distributing copyrighted works is
I sort of regretted downloading massive rom packs for retro consoles because I never actually get round to playing them. it’s so hard to settle on one when there’s so much choice. But now with the recent crack downs on ROM sites I’m really glad I’ve preserved them. Hoard away friend!
Roms don’t count!
I’ve done that with the Sharp X68000, I’ve downloaded the entire collection it has (~18GB). Do I know Japanese? No. Have I even checked if an emulator works with them? Also no. But it’s there
18gb is nothing at the end of the day. Roms are small. It’s the same thing with my 50gib font collection, its a lot of fonts, but little file size
I downloaded the entire SNES catalogue when I was overseas; so glad I did that then as it’s become much more difficult over the past few years.
II recently tried to download ISOs of my PS1 & 2 collections so that I could play them on my Steam Deck, and found out that at it’s actually a lot faster and easier just to to them myself - as whatever sources I can find are either dead links, or download so slowly that the connection is likely to time out before the download completes.
Use a 256 GB ssd Good luck hoarding on that rice grain
And this is why I consider SSDs to be a downgrade compared to HDDs lol
Aside from using the word ‘consume’…
I don’t like hoarding. I become so isolated from choice that I can’t enjoy anything I’ve ever acquired. I’ve always gotten what I wanted because I wanted it, not because everyone else has gotten it and not because it’s just to take up data on my drives. What I have now currently anyways, will sustain me for days to months and getting more of it will not make it better. It’ll just bring oversaturation and I’ll be too isolated again to bother.
No issue hoarding if you’re seeding