StevenWhiting.com

https://docs.microsoft.com/en-us/sysinternals/downloads/sysinternals-suite

Another old one from the XP days and I believe this was Outlook 2010.

Two users would kept getting this Outlook popup every few minutes. Very helpful message “A program”. Would be good if it could report what program had invoked it.

So I fire up Process Monitor and take a trace. Once I stop the capture I take a look. I assume it’s a not registry issue so I filter all reg entries out.

You can quickly do this by just clicking the registry icon

This now filters out all the reg entries in the trace.

I then filter out just Outlook to see if there is anything obvious and see these entries.

I pick the first TCP entry and press CTRL+B to bookmark it. I then go and turn the Outlook filter off so all other entries reappear. I do this to see what is going on around that time other than Outlook and I bookmark it in case I lose my place.

This reveals the app that appears to be invoking this and causing the popup

The piece of Crapita (sorry I mean Capita https://en.wikipedia.org/wiki/Capita) software appears to be reference the Outlook Object Library and then I see Outlook trying to connect to the exchange server. These all correspond with the pop-up. Speaking to the person who supported the Crapita app they discovered there was an issue on the server that was causing CapitaEVForms to do this. Once disabled the message stopped.

It appears if an external app attempts to invoke Outlook without permission or as in this case, access the address book without being “Trusted” then Outlook pops up a warning in case the access was malicious.

This is an old case from XP days, back in 2011. And a perfect example of David Soloman’s saying “Check Process Explorer AND Process Monitor”. It’s possible the cause will show in one but not the other as happened in this incident.

We had several users that would try to access network shares and then Explorer would hang for minutes. Eventually populating all the files that were on the share and giving the user access. This wasn’t always happening so when it did, instead of giving me time to troubleshoot I was ordered “Just rebuild the machine”.

Annoying. That isn’t fixing the issue, it’s just delaying it until it happens again. It was deemed quicker than finding out what was causing the issue. I disagreed. Having kit sat on the shelf waiting to be swapped out was all fine and good, but it still wasted their time. As they had to login to a new machine, set other settings up, wait for updates if the standby machine hadn’t been update recently etc.

The only troubleshooting I’d been given time to do was using Process Explorer. I could see Explorer running at 50% every time the issue happened. But why?

FINALLY, the day came where my manager had the issue and she was going on leave for 2 weeks. “Alright, use my laptop to see what you can find”.

Did you read that? Laptop, that was a key, all the users affected were on laptops. Hmm.

So when a process starts its made up of threads and stacks. So although it looks like “Explorer” is running at 50% and Task Manager showed this, Task Manager won’t show you the threads of stacks. I was able to reliably recreate the issue, finally so ran Process Monitor. Nothing. Just filtering Explorer at the time didn’t show anything obvious.

So I turned back to Process Explorer. Let’s look at the threads. Bingo.

A .dll that isn’t explorer.exe is running within the thread and it’s actually that .dll that is running at 50% NOT explorer.exe as crappy Task Manager shows. This is why Task Manager, even in Windows 10, is pretty useless.

PGP. That is familiar. That’s are encryption software we use on the laptops. We see this by clicking Module which takes you to the .dll’s properties.

So is this .dll important? We can’t not encrypt the laptops. Maybe there’s a patch. Despite it showing as PGP Corporation, Symantec had already bought them out at that point. So I did a Google search. And we find this

https://support.symantec.com/en_US/article.TECH149635.html

It states there are rare issues where this .dll can cause an issue and the work around is to unregister the .dll

Is that safe? Yes, all the thing did was search network drives for files that had been encrypted with PGP so it can then change the icon to show it’s encrypted. Pointless!

So any user that then ran into the issue I’d connect to their machine, run CMD as admin, then run the unregister command

regsvr32 /u PGPfsshl.dll

And that’s it. Explorer would no longer hang for minutes and the user could carrying on with their work. All within 5mins. 5mins compared to swapping out their laptop with a replacement. And that is why it is helpful to give your engineers time to find the cause of a problem and then a solution or work around. As most often, that can be a quicker fix that just a rebuild.

So a new addition to the software used at work would allow you to export files from it to a share. The user that was testing this was getting an export fail with cannot be accessed. But it was reported that user had full control access to the folder share.

The clue, really, is in the error dialogue box but appears not even the developers spotted it or even wasn’t aware of the specific reason why this would fail.

So I take a trace to see what is going on (I didn’t pay close enough attention to the error dialogue box myself, so even I missed it).

Anyway. Filtering is your friend in Process Monitor. You can collect so much info in a trace it can be overwhelming. I’ve learnt all I know from watching Mark Russinovich’s Case of the Unexplained videos he’s done over the years over and over. Every watch I see or learn something different.

I’ve put a few together on my YouTube channel as Microsoft lost some of the old ones and these need to be preserved.

Also others to watch are Aaron Margosis and old videos, if you can find them, from David Solomon. David and Mark did a great series way back when called “Sysinternals Video Library”. This was back when Sysinternals was still its own company and before Microsoft bought them out, so also before Mark started to work for Microsoft. Although most of the tools mentioned are obsolete along with the OS’ mentioned (Filemon and Regmon, which turned into Process Monitor), a lot of the info is still very useful.

Mark and David were kind enough to let me upload the library set to my YouTube channel so they are never lost.

Back to the point. So filters. A typical trace can run into the millions of events.

You want to capture everything because then you can filter. If you filtered before you could miss the very issue that is causing the crash or problem.

In this trace we assume it’s something to do with the app we’re in so we filter this.

So CRTL+L for filter

Here we choose Process Name “is” then choose the Process from the list. The only processes that will appear are the ones that were running at the time of the trace. Once the process is chosen we click Add.

The green tick means all other traces will now be hidden and we’ll only see stuff related to DocumotiveCapture.

Or the quick way is to find DocumotiveCapture in the trace, right click and “Include” the trace you want to filter.

Be aware. This will filter just DocumotiveCapture. If you then want to see other processes but not all, you’ll need to go into the Process Monitor Filter and then add each process you want to include.

So we are down from 600k showing to 108k

Then I like to run a results filter. It is a quick way to see if there is anything obvious in the trace, such as Access Denied, Network Path Not Found etc.

We choose Tools, Count Occurrences.

We choose Result and we click Count.

And straight away we can see 11 Access Denied. If we double click this, Process Monitor will automatically create a filter for Access Denied.

We see some registry keys but these don’t look like they could be the issue. And also a create file, but this is for just an icon in the Windows directory which is obvious the user shouldn’t have access to.

Then we see it in the next two lines. The reason for the error. Access Denied. But why? The user has full control over this share on the server. So why are they getting Access Denied.

This is the key

Only admins have permissions to these types of shares. A share with a $ sign at the end means it’s a hidden share. Not all $ shares are admin only. But all shares that point to a drive letter like c$, d$, e$ are all admin shares. And ONLY admins can access them. So even if you have permissions, as a normal user, to the Scans folder in this case, in this instance, as happened here, you’d get Access Denied. And that was it. The export function was hard coded to point to this e$ admin share so was always going to fail.

This was reported back to the developers who changed this to just point to the normal FQDN that the user had permissions to and that was it. Now exporting worked.

Gonna try and start a new series of Case of the Unexplained notes, much like Mark Russinovich’s, exactly like Mark’s infact 🙂

These are issues I’ve come across and fixed over the years with his wonderful Sysinternal tools. He’s been a part of Microsoft for years now since they bought out Sysinternals but the tools have continued to be developed.

Back to the case.

The users would load a database but it would pause for several seconds which eventually made it just unusable.

Looking in the Process monitor trace you could see the database being read over the network

During the trace it gets read several times with long intervals

Looking at the size of the .mdb file (which I don’t have an image of) it was discovered it was over 1GB. Then looking at the switch the users were connecting through, it was only 100MB. So that must be it. Replaced the switch with a 1GB switch and all was well again.

Use PSEXEC -s

Normally used for remote process running but can be used to run local as well

https://technet.microsoft.com/en-us/sysinternals/psexec.aspx

http://msdn.microsoft.com/en-us/windows/hardware/gg463028

For coders, crash dump analysis or use with Process Monitor and Process Explorer from Sysinternals.

srv*c:\symbols\publics*http://msdl.microsoft.com/download/symbols