hckrnws
Computational chemistry, Locally? My rant on SSH
by crescit_eundo
by crescit_eundo
I also wonder how they didn't come across X Forwarding at all? When i used a remote machine for my thesis that was one of the first things i set up, just so i could do small experiments on GPU and look at results more easily.
Basically every issue they seem to have is solved by using VSCode + Remote development (Transferring files, Customizing hotkeys, looking at outputs)
>And as for compute, I also realised that there are many small tasks that can actually be run on my machine
Yes, if you have a $3000 Computer, you can do many things that other people would need to use remote compute for. Depending on the "Cluster" they are talking about, the mac might even have better specs lol
[deleted]
Sad that it's necessary, but it's still a shitty environment. That employers foist it on their employees doesn't make it better.
>I also wonder how they didn't come across X Forwarding at all?
Xforwarding over VPN+residential broadband sounds like a great way to ...not use any sort of gui app.
During COVID I had the wonderful opportunity to remote into a work computer that had a Matlab license and watch the screen slowly paint in. Each keystroke took multiple seconds to display for me.
Also just a hot tip, if you’re able to keep any always-only PC on a university subnet (your PI should have some form of a server) you can use tailscale as a jump proxy and bypass the whole VPN dance.
I like being able to control those two variables separately.
The security team wouldn't budge things up to 1 day even.
When you have 2 or 3 factors on every tool -- even things like prometheus. It gets excessive.
Engine of a lawnmower, brakes of a Rolls-Royce, etc.
- Copy-paste link to browser
- Wait several minutes while page load hangs
- Click refresh a couple of times, page finally loads
- Type in work email
- Redirect to work portal, type in work email again
- Asked for work password: open KeepassXC and type in Keepass database password
- KeepassXC additionally demands that I enter my laptop PIN
- Copy work password into work portal
- Asked for Authenticator code: go look for phone
- Find phone, load up authenticator app, type into work portal
- Wait several minutes while page load hangs
- Click refresh a couple of times, eventually get code to paste back into Claude session.
By the description it appears computational chemistry is stuck in batch processing era. They are yet to discover their equivalent to REPL tight feedback loop workflow.
With a bit of automation you can work locally, submit big jobs to the remote cluster, and transfer the data back automatically. It can help to have an always-on computer that you control. For me this is:
[macbook]*tailscale*[my server]---ssh---[cluster]
This solves the problems of pushing data back to a laptop that might be turned off or moving location, on 4G etc. While a job is running, output files start streaming back with rsync.
Working on a cluster can be a pleasure if you use the right tools.
it doesn't handle the VPN/2FA friction I guess, but I'd think unless most work can be done locally it would be better to invest in making cluster work more convenient
It's really, really hard to shrink the problem and troubleshoot with large, opaque, containerized chunks of functionality and services.
Can I just connect to a DB in a terminal and run some SQL to figure out the problem?
Why, no: no, you cannot.
man 5 ssh_config
/ControlMasterLove the ability to just forward one or several applications directly instead of the whole desktop environment. Ability to resume connections are also a godsend.
Seems like that usecase isn't given a lot of effort nowadays.
I agree that X forwarding is a terrible way for GUI apps, but the OP read like they were just looking at logs or plots/visualizations without much interactivity. X forwarding is good enough for that (At least good enough to give it a try over scping files back and forth)
This is an excellent way to have yr. campus infosec ppl shut the network port on said server, and then take the hardware to get its OS install scrutinized, and get your accounts locked in the process.
Don't do this.
Universities use all kinds of hard to setup stuff which are often custom built. Slurm configs, custom configured HPC servers etc. All of this pays off though if you need to do heavy computations which a local machine won't do.
IMO it would be better to make Claude build out something that makes interacting with the HPC much easier. A web UI, Tailscale whatever.
Not the best security practice but thought you should know!
The only two startups I'm aware of who have managed to make compchem-as-a-service a viable business are Mat3ra and Materials Square. Both are a little rough in their own ways (to be expected from startups), but it's still interesting to see some of this move beyond university clusters.
I wouldn't call it "stuck," since it's one of the few fields where batch processing makes sense.
You generally run your jobs on a supercomputer that's being shared by a few hundred other people. You need a scheduling system to allocate resources (e.g., 5 jobs requesting 10 nodes for 8 hours, 1000 jobs requesting 0.5 node for 5 minutes, etc).
It's hard to tighten the workflow when the average calculation takes 2-3 hours of time.
You will see a lot of similar procedures in, say, AI model training and refinement. Those depend on batch jobs as well, pretty heavily. So the batch process isn't really restricted to the weirdness of academia. My source there is actually working on the systems doing training for frontier models...
(Please forgive some of my hand-waviness or vagueness, it's been nearly 20 years since I last touched a lot of this).