A while back I was developing a workflow scheduler. However the run duration kept coming back negative. The runs kept finishing like 15 seconds before they started. Eventually I figured out why; the company had configured a policy to use their custom NTP server, but it was 16.7 seconds off. So my Windows laptop’s time was in the future, but the WSL VM (which ran the program) used another NTP server and was correct. In the end I just added a bodge that made the program request the time from an external server on startup to calculate an offset.
67
23 comments
log in to comment.
i was just a user, but i debugged a weird issue where one of those blue light reduction programs would start flickering and inverting day and night time during june.
turns out the dev had made the assumption that the sun always rises before it sets, and i was far enough north that the sun would start setting after midnight.
youtu.be/-5wpm-gesOY
Such a classic.
Ooh, interesting one! 😄
i was trying to figure out why ssh wasnt working only to realize I was using the wrong username. That took about 2 hours of time to realize as I was bashing my head at the wall while looking at the pubkey signature being the same on both machines.
I found an issue in a software used by government employees.
A request would fail if the server and the client had different years in their system time which could happen if they were located in different time zones.
After a few calculations I asked: “Hey, what are the chances a government employee would do this important task between 6pm and midnight on December 31st?”
We decided not to prioritize the issue.
You realise that’s going to end up in the launch process for a nuke somehow.
I guess “this important task” could refer to firing a nuke, which to be fair is unlikely to happen at that moment in time.
Govt bureaucracy. It could be scheduling a janitor and still somehow end up in the checklist.
I’ve been a dev for years and had my fair share of sillyness… can’t recall which was the most silly, but one very recent one was trying to debug a weird error. So I opened the VS solution did a search for the string that spat out the error. Found it in a catch clause and stabbed that function with breakpoints, and went off to test.
Hours later (too many to confess) after trying everything, I found out why I couldn’t get it to stop on the breakpoint. I had opened the wrong project, for the wrong app :O
It was an unlucky coincidence that it had the same literal error trace in code!
Wrote 1 instead of i in a for loop and wasn’t allowed to use a nice font like hack
Two variables named somethingl and somethingI that had three grown developers scratching their head why the wrong variable was getting updated. This was back in the day when intellisense and colouring was way less advanced.
An actual compiler bug that wrote into the wrong register, went all the way down to assembler to realise the issue. This was on the early Xbox 360 (PowerPC) compiler.
two days for an
=instead of an==This was on a Fortran project that had no encapsulation and way too many global variables.
Suddenly, one particular calculation was failing, and not in a floating point rounding error kind of way. After setting up several scenarios to generate various input values, I determined that Fortran’s absolute value function was returning the wrong results.
I couldn’t understand why, so I went to the Lead Architect. He poked at it for about four minutes and did some searches, and found where someone else had defined a global variable named ABS that was masking the built-in abs function. The function call argument [e.g. abs(3)] was being compiled as an array index on this ABS variable [which was not an array, so yay, undefined behaviour].
Used a niche language for specialised hardware that ignores the case values in switch statements.
Switch x { (x=0 executes this code) Case 123: (x=1 executes this code) Case 321: (x=2 executes this code) … }
Also Java 7 timer object locks up when windows changes the system clock backwards during DST.
IT department really wanted to use their snake oil antivirus enterprise super secure hackproof™ software everywhere.
So it was installed on the CI servers.
Fastforward a few months and we were wondering why one build agent was no longer picking up jobs.
After a day of debugging we found out that a certain snake oil sofware tampered with a single machine internal TCP conncetion and somehow fucked it up in a way that all packets in one direction just vanished.
Disabling the snake oil software magically fixed the problem.
Should a CI server even have access to the Internet and need antivirus software? Maven access I guess.
Whenever I book a flight with Air France, I have to enter our birth dates off by one day. We’re in Canada, and the Air France servers are most likely in…France, or at least using a french time zone.
Anyways, I figure that someone decided to use some form of LocalTime midnight for the date. We’re 6 or 7 hours later, so when it translates it to French time, it’s a different day.
So, not a bug that I fixed, but one that I work around. It’s been this way for years, too.
Took a course during uni on computer graphics and one assignment I handed in needed correction, so I spent the better part of a summer trying to debug and find the flaw. Difficult to debug the graphics pipeline back then and with basically no tutors on hand during vacation times.
Turns there was nothing wrong with my thinking and program logic, I had just accidentally written two adjacent lines in the wrong order.
I transposed them, everything worked.
I was both relieved and pissed off at the same time. So clumsy.
Was told to test a new database migration as they updated from mainframe to a current version of Oracle. First I was to test data integrity. Wrote a python script to pull from the old db and verify that record matched the new. Found sooooo many bugs, sent it to the dev. Half were real bugs, half were bugs with my script. Turns out they didn’t actually want a 1:1 on the new db they were going to change things cuz they could. Lots of back and forth and by the end my python script essentially mirrored the production code.
Then they wanted me to load test the new system. I wrote a quick script to query N rows from the db with X threads and blast the new service. Didn’t take long to write, presented my finished script and plan to pull in manual testers to also run my script as needed.
Dev was cocky because they’d setup auto scaling on AWS. Manager was concerned I’d taken time to write a script instead of just spamming the same message over and over with a tool like gatling.
First run I took down the system very quickly, like within a minute with just my laptop. Note, this was a system that was planned to handle 100s of millions of requests an hour. Dev was impressed, turned out to be a auto scaling misconfiguration. Second time it took a few mins but it died again. Config needed some more tweaks. Third time I took it down because of some queueing issue that raised it’s ugly head. Fourth time I took it down they found a bug in the code. Fifth, Sixth, and Seventh time I needed the manual tester’s help, but the devs were scratching their heads trying to figure out how I was taking them out in 5mins of sustained load. Took two teams of devs scratching their heads and several itterations through bringing the system down and even two separate devs walked through my code to see what on earth I was doing. To top it off it wouldn’t just take down the microservice, it would crash our entire AWS infrastructure for the company. Not just our project but all our aws projects in the building went down.
They finally found the problem. The silly little problem. Remember that data I pulled from the db and blasted them with? Well, we hit a poision pill situation. One of the records turns out was a manual tester’s favorite to use when running one of his tests. That one record had 30,000 attached sub-records. Everytime that tester ran that test case he just kept adding a sub-record to it. Dude didn’t know he was slowly crafting such a massive data bomb all those years that would take down all our AWS servers.
They had also fixed the problem with one service crashing the others. And that my friends is why you never skimp on good automated testing. When it went live there was no major issues or outages, though we did let it soak in parallel with prod for weeks to make sure we trusted the results.
That I have a wetware problem.
When I'm debugging, I usually forget the nature of a bug when it is fixed. The more trivial it is, the quicker I seem to forget.
Something does trickle down to my unconscious because I would recognise getting the same error message over and over and that doesn't seem to happen often, but I could not tell you a single bug in the last thing I wrote, even though I know there were plenty of them.
What really makes this weird is that I sometimes become very upset - we're talking wailing and gnashing of teeth here - when I can't find a bug, and you'd think I'd remember those, yet once I do finally track it down and get over feeling angry and stupid, the nature of the bug still disappears from conscious memory.
Now, maybe I am learning lessons because of the infrequent recurrence of the same mistakes, but I'd like to be able to think about them later, and I can't do that.
They did a 400 table join to return ‘true’ to test to see if they connected to the database; it took 48 minutes to run. Printed out, it took up 38 sheets of paper.
I replaced it with a dozen sequence checks and individual select statements for each table; it took 3 seconds to run. And it could accurately identify which tables and columns are missing. Printed out, it fit on a single page.
I had a project I was working on a while back. I kept testing it and trying to find the bug that was causing my output to be completely wrong. It took me about 2 days to finally find it:
I had a boolean set to false, when it should have been set to true.
That one little bug completely broke my sorting algorithm. After changing it, it worked fine. :/