Category Archives: Rants

Exchange 2010 SP2 – A Warning

Posted by on 16/12/2011 2 comments

You may remember my post a while back about issues with applying Exchange 2010 SP1 in situations where you were using Group Policy to control Powershell Execution policies. Specifically, this issue occurs because the Group Policy setting uses the WMI service to enforce the Execution Policy and as part of the Exchange install/upgrade process, the WMI service is stopped, causing the Execution Policy to revert to Restricted and the following error to pop up and the install to fail:

The following error was generated when "$error.Clear();
& $RoleBinPath\ServiceControl.ps1 EnableServices Critical
" was run: "AuthorizationManager check failed.".

Well it turns out that this still applies with Exchange 2010 SP2 in exactly the same fashion.

The (well, A) relevant KB article is here but the “workaround” is a bit half-assed to be honest and you’re much better off just disabling the associated Group Policy setting and configuring the Execution Policy locally (with set-executionpolicy) to either AllSigned,RemoteSigned or Unrestricted for the duration of the upgrade.

Why Microsoft cannot add installer logic to check for this possibility, especially given how long it’s been a potential problem, is beyond me but then I’m not an Exchange developer.

Spamming Bastards: Part Deux

Posted by on 12/12/2011 4 comments

My spam Blocklist has been slowly growing since I created it in order to stem the tide of comment spam coming from Nobis/Ubiquity Server-owned address blocks. Ultimately I made the choice to block all the netblocks that they had allocated from ARIN and that seemed to have worked, up until today when I started getting comment spam from some brand new Nobis/Ubiquity addresses.

It would seem that they’ve got themselves a netblock from RIPE and started using that for spamming as well; the range in question is 176.31.50.64/27 but given their apparent dedication to illegal activity, it wouldn’t surprise me if others start popping up here and there as well. Thankfully, the relative scarcity of available IPv4 blocks is making it much tougher for these spamming fuckers to evade blocking mechanisms without resorting to botnets.

That said, when you’re getting more than 10 times as many spam comments as legitimate ones, it doesn’t exactly fill you with confidence that we’ll ever get a real handle on the problem.

What The Hell, Adobe?

Posted by on 08/12/2011 No comments

OK, so Adobe’s Flash Player downloads have always been crappy; half the time you can’t find the download for non-IE browsers and the other half of the time you’re forced to use the Adobe Download Manager to get it, but I think they’ve managed to outdo themselves now.

If you go to download Flash Player from the Adobe website (http://get.adobe.com/flashplayer) and you’re not running IE, you’re presented with some drop-down boxes asking you to pick your OS and Browser (IE or Other). If you pick “Windows 7 (64-bit)” – as well you might if that’s the OS you’re running – then you get given the 64-bit version of Flash Player, which won’t work for 95% of users and won’t be clear why, so you have to pick “Windows 7 (32-bit)”, obviously.

You then download the installer and run it, at the end of which, the installer deletes itself, presumably because they realise that it’s such an insecure piece of crap that by the time you come to install it on a 2nd machine 3 minutes later, there’s already a new version out.

To make matters worse, the MSI distribution points (such as http://www.adobe.com/go/full_flashplayer_win_msi) are still Flash 10.x instead of 11 and the old http://fpdownload.adobe.com/get/flashplayer/current/ links now 404.

I’m already trying to get rid of Flash Player, I’m using Youtube’s HTML5 trial and making use of alternative options where ever possible, but it’s like Adobe is actively trying to help me stop using it.

Fuck You Adobe.

ProTip: Outlook 2003 & Public Folders

Posted by on 09/08/2011 No comments

As you probably know, Outlook 2003 and older use Exchange Public Folders for their Free/Busy data and Offline Address Book. You may also know that Exchange 2007 and Exchange 2010 have deprecated Public Folders in favour of an HTTP-based distribution method for the data, which Outlook 2007 & 2010 fully support.

One could then reasonably conclude that removing public folders in an Exchange 2007 or Exchange 2010 environment might negatively affect Outlook 2003 and older clients. This is technically correct, though the reality is slightly more dramatic; it completely blocks Pre-Outlook 2007 clients from connecting to your Exchange servers and presents the users with a handy “Your administrator has blocked this version of Outlook from connecting” message.

What this all means is that if your support team have spent the last year lying to you every time you’ve asked them how they’re progressing on upgrading all the remaining Outlook 2003 installs to 2007/2010 then when you remove the last public folder store from your Exchange environment you’ll suddenly have hundreds of people whose can no longer access their emails. This, for some reason, upsets them.

The moral of this story is that you should never trust what people tell you, because they’re almost always lying bastards, and make sure you verify the information for yourself before making any changes. Yes, it’s a lot of work and no, you shouldn’t have to do it, but it’s always you that gets the flak when it all hits the fan.

Spamming Bastards

Posted by on 20/06/2011 2 comments

Given that my blog is relatively low traffic, it’s remarkable just how many spam comments and hacking attempts I log daily. A good 50% or more of all the spam comments I get originate from the same place: Ubiquity Server Solutions/Nobis Technology Group, who share a couple of overlapping IP ranges and are somewhat notorious if my brief Googling is anything to go by. I’m a big fan of Hanlon’s Razor, but in this case I’m really not sure either way.

So, as of today, their entire ranges are blacklisted:

Deny from 173.208.100.0/22 #Ubiquity Server Solutions
Deny from 108.62.0.0/16 #Nobis Technology Group

I don’t like having to block entire /16 ranges because I know there are bound to be false positives in there somewhere, but frankly it’s the only way to make things manageable right now.

I expect to see my error.log grow exponentially over the next few days.

Update: And another range of theirs that was still spamming me…

Deny from 173.234.0.0/16 #Nobis Technology Group

Update: And yet another…

Deny from 23.19.0.0/16 #Nobis Technology Group

Update: Guess who…

Deny from 64.120.0.0/17 #Nobis Technology Group

Update: Right, let’s make this simple; courtesy of ARIN’s WHOIS Database

#All Nobis/Ubiquity ARIN Netblocks
Deny from 70.32.32.0/20
Deny from 67.201.48.0/23
Deny from 72.37.145.0/24
Deny from 173.208.0.0/17
Deny from 69.174.60.0/22
Deny from 174.34.128.0/18
Deny from 173.234.0.0/16
Deny from 108.62.0.0/16
Deny from 72.37.224.0/21
Deny from 23.19.0.0/16
Deny from 72.37.237.0/24
Deny from 72.37.218.0/23
Deny from 72.37.222.0/23
Deny from 72.37.221.0/24
Deny from 67.201.0.0/21
Deny from 72.37.242.0/23
Deny from 67.201.40.0/24
Deny from 72.37.246.0/23
Deny from 216.6.224.0/20
Deny from 72.37.204.0/24
Deny from 69.147.224.0/23
Deny from 64.120.0.0/17

Update: My complete comment spam blocklist is now available here.

Despair

Posted by on 12/06/2011 No comments
  • Who has 1,450 login scripts (For 3,500 users at that)?
  • Who puts account passwords in the account’s description field? (for anyone who doesn’t know why this is appallingly bad, try this PowerShell as a regular domain user from a box with the Win7/2008R2 RSAT tools installed: ipmo activedirectory;get-aduser -filter * -Properties description, other scripting languages are available)
  • Who doesn’t use Domain Admins for their domain administrators and instead uses Account Operators, Administrators, Network Configuration Operators, Remote Desktop Users & Server Operators? (Not for granular permissions, all of them as a replacement for Domain Admins membership)
  • Who leaves 5,500 computer accounts, including servers, in the Computers container in AD? (ProTip: You can’t link GPOs to the “Computers” container in AD)
  • Who doesn’t have any actual DHCP servers in their Authorised DHCP Servers list? (Not even sure how you manage this one)

This isn’t just bad practice, this is years of dedicated training and substantial investment in bad practice…

I’m going to go curl up in a corner and cry for a while now.

This Is Not A Joke

Posted by on 23/05/2011 No comments

I have previously completed some work for a mid-sized organization and some of the things I’ve come across in the few shorts weeks I’ve had access to their systems are, frankly, astonishing. I’ve barely even scratched the surface and we’re already well into “so bad it’s not even wrong” territory here; a few examples:

  • Critical Servers not under warranty
  • Critical Servers not backed up
  • Servers backed up to disks on the same disk array as the live data
  • Backups taken to tape and then left in the tape loader indefinitely
  • Servers not patched, ever
  • Antivirus not installed on servers, or installed but disabled “for performance reasons”
  • Server hardware/software not monitored
  • Speed/Duplex on all network interfaces set manually
  • Public address ranges for internal network addressing
  • A single broadcast domain for all devices on the network
  • 120m+ Ethernet cable runs
  • Cat3 cable runs within the core infrastructure (undocumented)
  • New user credentials sent by email, via an externally hosted mail system, to user’s line manager
  • All IT Staff granted unaudited access to the entirety of the file servers
  • All IT Staff local admins on all servers

And that’s just the stuff I’ve found so far and haven’t already repressed. I honestly have no idea where to begin, if there were such a thing as a Worst Practice Guide they’d have not only followed it to the letter, but added their own extensive appendices as well.

Exchange 2010 EMC and Non-Existent Servers

Posted by on 27/04/2011 No comments

It would appear that the Exchange 2010 EMC isn’t particularly bright; when you launch it, it picks a CAS to connect to from AD. This is fine.

However, should that server cease to exist, by which I mean Exchange is uninstalled and it is properly decommissioned, then the EMC will continue to try and connect to it. Even after the connection fails, it’ll keep on merrily plugging away at the non-existent server, never considering that there are probably other servers it could try.

This is very annoying and seemingly very stupid behaviour. To work around it, close the EMC, fire up your registry editor of choice, locate the following key: HKCU\Software\Microsoft\Exchangeserver\v14\AdminTools\ and delete the NodeStructureSettings value. This will reset the EMC and cause it to pick a new CAS to connect to; it may also affect other settings that you’ve changed in the console.

Another option is to close the EMC, navigate to C:\users\<username>\AppData\Roaming\Microsoft\MMC\ and delete the Exchange Management Console file. This will also reset the EMC and will definitely reset any customisations you’ve made to the console.

Why you should ever have to do this is something of a mystery to me, perhaps Microsoft just never expected anyone to decommission an Exchange server once it was built.

Wherein Our Hero Battles Exchange 2007

Posted by on 30/03/2011 1 comment

So, you’ve got an Exchange 2007 CCR Cluster set up and all is well in the world; your data is safely replicated offsite so that in the event of a disaster, you can have your users back up and emailing in the time it takes a DNS record to update.

But then, disaster! A different disaster to the previously mentioned one, obviously, because this disaster causes a cluster failover and the connection between nodes is down for just long enough that they get out of sync and require a reseed to fix.

At this point I’d like to jump off on a slight tangent to bemoan the inconsistency with which Exchange 2007 handles the interruption of replication traffic. On the one hand, you can shut down one node for a couple of hours and when you bring it back up again replication resumes quite happily, but on the other hand if you have 5 minutes of iffy network connectivity, suddenly the databases* are irrevocably out of sync and need to be reseeded.

Anyway, you’re not too concerned by this turn of events because, while a reseed of your ~90Gb database takes a few hours it’s not like the cluster is going to fail back while you’ve got databases in an inconsistent state, is it?

Well, it shouldn’t have happened, but it did; the bloody thing failed back while halfway through reseeding the database and then, obviously, couldn’t mount it at the other end. This posed something of a problem, because Move-ClusteredMailboxServer (or the GUI equivalent) gets upset when your databases aren’t in sync and refuses to let you fail over and Restore-StorageGroupCopy would have forcibly mounted the database sans up-to-date logs and effectively reverted it to the state it was in before it all failed over the first time, binning a lot of emails in the process.

Thankfully, Move-ClusteredMailboxServer has a very handy -IgnoreDismounted option which allows you, when you’re really sure, to skip all replication health checks on Dismounted databases, allowing you to fail the server over and remount the (more) up-to-date version of the database, whereupon you can attempt to re-reseed it. So, if you find yourself in a similar quandary, with your databases all out of sync and at risk of losing hours or even days worth of data, before starting a restore from tape & printing your CV, you can always try: Move-ClusteredMailboxServer -Identity <CCR Cluster Name> -TargetMachine <Target Cluster Node> -IgnoreDismounted Just remember that there’s a good chance of at least some data loss, but if you’re in a position to need to use it, the alternative is probably a lot of data loss so it’s a risk that might be worth taking.

* I know that technically Exchange 2007 CCR replicates at the Storage Group level rather than Database level, but as you can only have a single database in a CCR replicated Storage Group and “database” is easier to type, I’ve used it instead.

So, The iPlayer For Android…

Posted by on 11/02/2011 No comments

Well, the BBC iPlayer for Android has been released and I’m really disappointed.

Before the app I could go to the iPlayer website on my phone and stream recorded TV and Radio programs over 3G or Wi-Fi; no Live streaming, but otherwise pretty good.

With the app, however, I can’t stream anything unless I’m on a Wi-Fi connection and as far as I can see there’s no way to override it, so the fact that they now offer live streams is all but worthless as if I’m somewhere with Wi-Fi I’m usually somewhere with a TV or Radio. What’s even worse is that they’ve now applied the same fucking policy to the mobile version of the iPlayer website too, so I can’t even stream *that* over 3G any more.

Why have they done it? No idea, but it’s bloody stupid. By all means make it default to Wi-Fi only to stop all the idiots complaining when they stream the entire Eastenders back catalogue over 3G and run up for £3,000 phone bill, but I’m not one of those idiots; I want live 3G streaming and at the very least I want my recorded 3G streaming back. I can only imagine that the mobile networks threatened to block all iPlayer traffic if the BBC released their app with 3G support, because we all know that’s easier than actually upgrading your networks to support demand.

Oh, and it can’t run in the background either, or if your phone switches off the screen.

It’s been removed from my phone after a grand total of 8 minutes. Not happy.

Update: According to the FAQ here “[they] are working to make the service available on 3G networks in a future release of the BBC iPlayer Android App.” So that’s all OK then…