SOLR performance slow disk IO problem

Problem

My colleagues and I have been having significant performance issues with SOLR which I know is being used by many other organisations without any issues on very large indexes.

Our SOLR has indexed approximately 5 million documents. The contents of the documents are also set as stored fields. Most of the documents are quite large also.

The SOLR data directory is around 500 Giga Bytes in size.

Our SOLR version is 1.4, although upgrading to 4.1 made no difference.

There is only a single shard and single core.

SOLR server is on a Windows Server VM with 24GB RAM and 8 CPU.

We use VMWare as our virtualisation platform.

Our Java web application which runs in Tomcat utilises the SOLRj client to make requests to SOLR. We request 1,000 results from SOLR on user searches as the permissions to the documents in our SOLR index are handle by the database server which SOLR has no visibility to. We need to request 1,000 records from SOLR so that we can then filter the results based on user's access and then only show the top 100 results which they have access to.

Our storage solution is a 10GB iSCSI DELL Equalogic SAN which we know is capable of high throughout and high IOPs.

We were finding that there would be a significant delay in situations where SOLR had to go to disk to load index data that was not cache in memory. This delay could be anything up to two minutes at times.

After doing lots of research we found many other blogs and discussions on the Internet which would suggest how to tune and optimise SOLR. Some of which helped slightly, but we still seemed to have this underlying disk IO issue.

When SOLR was busy trying to get index data from disk we took a look at the SAN IO graphs to notice that SOLR was only capable of performing the random read of the index files at 6MB per second. We knew our SAN was capable of much more than this.

Solution

After trying many things to optimise this, the only solution that worked for us was to get our java application to make 10 concurrent requests to SOLR, each asking for 100 record result set instead of a single request for 1,000. This seemed to have a significant positive impact on the performance as now with multiple threads, SOLR could read data much faster from the storage array even though there is still only one single SOLR server with a single shard.

We are not 100% sure why we had to do this, but it worked for us. It could be because of any of the following reasons:

1. We use iSCSI to connect to our storage, which I have heard can have IO issues under certain single threaded circumstances.

2. We use Windows environment rather than UNIX.

3. May be a Windows / VMWare / DELL iSCSI specific issue.

4. Limitation of SOLRj client.

SVN Mirroring with VisualSVN

create blank destination repository in Visual SVN manager run following: svnsync init [destination repository URL] [source repository URL] From VisualSVN Manager, right click on new destination repository and select “All Tasks \ Manage Hooks”. Select “Pre-revision property change hook” and edit that – enter a couple of blank lines then click ok. Done. The hook will now be listed in bold indicating it exists. run following: svnsync --non-interactive sync [destination repository URL]

Number of rows and size of tables in MS SQL 2005

There are times when you need to know what in your database is taking up all the disk space.

Or you may just want to know which tables contain the most number of records.

The following Query in MS SQL 2005 will give you the answer:

CREATE TABLE #temp (
table_name sysname ,
row_count INT,
reserved_size VARCHAR(50),
data_size VARCHAR(50),
index_size VARCHAR(50),
unused_size VARCHAR(50))
SET NOCOUNT ON
INSERT #temp
EXEC sp_msforeachtable 'sp_spaceused ''?'''
SELECT a.table_name,
a.row_count,
COUNT(*) AS col_count,
a.data_size
FROM #temp a
INNER JOIN information_schema.columns b
ON a.table_name collate database_default
= b.table_name collate database_default
GROUP BY a.table_name, a.row_count, a.data_size
ORDER BY CAST(REPLACE(a.data_size, ' KB', '') AS integer) DESC
DROP TABLE #temp

Converting certificate from PFX to PEM

openssl pkcs12 -export -in abc.crt -inkey abc.key -out abc.pfx

Windows Firewall keeps turning on after adding to domain

After adding servers to Windows server 2008 domain controller the firewall keeps turning on afert each reboot.

This is because the domain is configured in this way.

To turn this off, on the domain controller, go to:

"Group Policy Management" ->
forest ->
Domains ->
relevant domain ->
Group Policy Object ->
Default Domain Policy.

Click right mouse button on the above and edit.

This will open up the Group Policy Management Editor.

expand:
Computer configuration ->
Policies ->
Administrative templates ->
Network ->
Network Connections ->
Windows Firewall ->
Domain Profile ->
Windows Firewall: Protect all network connections

Set the above to "Disabled"

HowTo: Firefox and Integrated Windows Authentication

As per original article:
http://markmonica.com/2007/11/20/firefox-and-integrated-windows-authentication/


Do you have an Intranet or a similar web site that requires the use of Integrated Windows Authentication? If so the default Firefox browser settings will always prompt you for a username and password first before accessing a site using Integrated Window Authentication.

Fortunately Firefox has the slick ability to easily modify it's configuration to use Integrated Windows Authentication.
How to configure Firefox

Open Firefox
In the address bar type: about:config
Firefox3.x and later requires you to agree that you will proceed with caution.
After the config page loads, in the filter box type: network.automatic
* Modify network.automatic-ntlm-auth.trusted-uris by double clicking the row and enter http://www.replacewithyoursite.com or http://your-intranet-server-name
* Multiple sites can be added by comma delimiting them such as http://www.replacewithyoursite.com, http://www.replacewithyourintranetsite.com

Package for Large Installs

If you are a network administrator that has a lot of installs to do, you can modify the Firefox installer.

Use a tool such as 7-zip to extract Firefox Setup 2.x.x.exe
Extract browser.xpi from the setup
Edit all.js contained in browser.xpi contained in binjreprefs
Modify the entries as in items 4 and 5 above
Re-package browser.xpi and use the extracted setup to install Firefox

* Special Notes

To specify all subdomains use .replacewithyoursite.com instead of www.replacewithyoursite.com, help.replacewithyoursite.com, pictures.replacewithyoursite.com

Managing mssql tempdb database and files

If tempdb is set to auto grow, then it will grow only to fill the disk it is in and no more.

So I have created a 1GB ram disk (http://memory.dataram.com/) and set the tempdb to be on the ram disk. This has significantly improved the performance of mssql although I thought tempdb is supposed to be in ram most of the time anyway!!!

The below allows you to see current details of tempdb and files and how they are set to grow:

SELECT
name AS FileName,
size*1.0/128 AS FileSizeinMB,
CASE max_size
WHEN 0 THEN 'Autogrowth is off.'
WHEN -1 THEN 'Autogrowth is on.'
ELSE 'Log file will grow to a maximum size of 2 TB.'
END,
growth AS 'GrowthValue',
'GrowthIncrement' =
CASE
WHEN growth = 0 THEN 'Size is fixed and will not grow.'
WHEN growth > 0 AND is_percent_growth = 0
THEN 'Growth value is in 8-KB pages.'
ELSE 'Growth value is a percentage.'
END
FROM tempdb.sys.database_files;
GO

An the below, lets you specify the exact size of the data and log files and how they grow. The values in the below are all in Mega Bytes.

ALTER DATABASE [tempdb] MODIFY FILE ( NAME = N'tempdev', SIZE = 900 , FILEGROWTH = 50)
go
ALTER DATABASE [tempdb] MODIFY FILE ( NAME = N'templog', SIZE = 50 , FILEGROWTH = 5)
go

If you want you can set FILEGROWTH to 0 which means it will not grow but this is not required as the tempdb will never grow past the desk space it lives on.

Making VIM display UTF-8 characters correctly

First you need to chose a font which has the characters you need to display

I have found this is a good one:

:set guifont=Consolas

Then you need to set encoding to utf-8

:set enc=utf-8

MS SQL performance tuning. (Identifying which query is taking the longest time)

Run the below query against a database to get very useful information about how long each query is taking and how many times it has been executed

SELECT TOP 100
qs.total_elapsed_time / qs.execution_count / 1000000.0 AS average_seconds,
qs.total_elapsed_time / 1000000.0 AS total_seconds,
qs.execution_count,
SUBSTRING (qt.text,qs.statement_start_offset/2,
(CASE WHEN qs.statement_end_offset = -1
THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2
ELSE qs.statement_end_offset END - qs.statement_start_offset)/2) AS individual_query,
o.name AS object_name,
DB_NAME(qt.dbid) AS database_name
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt
LEFT OUTER JOIN sys.objects o ON qt.objectid = o.object_id
WHERE qt.dbid = DB_ID()
ORDER BY average_seconds DESC;


You can run the below to clear the cache at any time:

DBCC FREEPROCCACHE

Moving MS SQL TEMPDB

If you need to move the location of the tempdb data or log files you can use the following:

This tells you where everything is now:

USE TempDB
GO
EXEC sp_helpfile
GO

This sets the new location for the data and log file. If you want you can move just one or both files.

USE master
GO
ALTER DATABASE TempDB MODIFY FILE
(NAME = tempdev, FILENAME = 'd:\newlocation\tempdb.mdf')
GO
ALTER DATABASE TempDB MODIFY FILE
(NAME = templog, FILENAME = 'e:\newlocation\templog.ldf')
GO

You must restart MS SQL service after this. This will automatically create the files in the correct location for you.

If you got the wrong path and the service is not starting then you need to follow the below steps:

Restart SQL Server with only the master database.
NET START MSSQLSERVER /f /T3608

Using SSMS or SQLCMD, execute the ALTER DATABASE commands to move the tempdb files.
ALTER DATABASE tempdb MODIFY FILE (NAME=tempdev, FILENAME='new path\file')
ALTER DATABASE tempdb MODIFY FILE (NAME=templog, FILENAME='new path\file')
(Repeat for all tempdb files if you have multiple data files)

Stop SQL Server and restart it normally.

Restricted SSH connection

If you want to allow a user to only execute certain commands on a server which they have autologin ssh access to, you can acheive by using the following.

The below shows how to only allow user to run svn specific commands via their ssh autologin access.

insert the following in the relevant users authorized_keys file ni the users home/.ssh

command="svnserve -t -r / --tunnel-user=username",no-port-forwarding,no-agent-forwarding,no-X11-forwarding ssh-dss AAAAB3....rp9vp9Q== username@hostname

Windows Server 2008, multiple IP addresses, primary IP, secondary IP confusion

When multiple IP addresses are assigned to a single Windows Server 2008 OS, the OS seems to randomly choose any one of the multiple IP addresses assigned as the primary IP address. Even though the GUI would suggest otherwise.

The problem with this is that if you have firewall rules allowing only the primary IP address of this server to other resources, requests from this server are going to be blocked as the target is not seeing the primary IP address you specified, it sees one of the other IP addresses instead.

This can also be a problem with mail servers as you may have specified SPF records which expect emails to be sent only from one IP address, but instead are being sent from another IP address on the same machine because of the same problem.

Solution is explained in below article.

http://support.microsoft.com/kb/2386184/

1. Find the correct patch for your OS, download, install and reboot. This patch will add "skipassource" capability to "netsh" command.

2. Remove all secondary IP addresses and add them not using the GUI, but using the following comand:

  
   
Netsh int ipv4 add address "Local Area Connection" 192.168.x.x 255.255.255.0 skipassource=true

The Skipassource=true causes that IP address never to be used as a primary IP address.

This should resolve the issue.

IIS features to enable in order to install MS SQL 2005 on Windows Server 2008

Before installing MS SQL 2005 on Windows Server 2008 make sure you enable features described in following Microsoft article:

http://support.microsoft.com/kb/920201

Windows Server 2008 Rsync dot folder problem

I had a simple script which contained a number of rsync commands to copy a number of folders from one machine to another.

For some reason when I execute the script, the target/destination server first creates a dot (.) folder and then copies the files in there!!!!


I was using vim to create my scripts on a windows machine.

I found that by changing the file format to a unix file format using command ":set ff=unix" and then saving the script and running it again, the files synchronised into the right folder.

Delete files older than a number of days

 If you want to delete all the files in D:\temp which are older than 30 days old then just run the following command:

forfiles /p "D:\temp" /S /M * /D -30 /C "cmd /c del /Q @path"

Windows Server 2008 losing IP address after reboot and cannot change Primary IP address at all

There is a bug in Windows Server 2008 which causes the server to randomly loose IP addresses after a reboot. Also for some reason I cannot remove/replace the primary IP address with any other IP address either.

You can see all the IP addresses that are meant to be assigned to the NIC in the GUI but when you do ipconfig you only see some of the IP addresses. If you try to ping the missing IPs, you will not get a response either.

There is a Microsoft patch will fix the problem with missing IP addresses.


KB Article Number(s): 982931


Request the hotfix from MS, install and reboot. By the way, the exe MS provides suggests that it is for VISTA only, but it is for Windows Server 2008 also.

Sadly the primary IP issue continued post hotfix and needed the removal of IP's, disable the card, delete the reg key (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Network   'Config' reg_binary), restart, enable the card, assign IP's and test

This did not work for me.

After reboot I still have missing primary IP and the only solution is to remove all IPs manually and the run a script which adds all the IPs back to the server with a number of the following:



netsh interface IP add address name="Local Area Connection" 192.168.1.20  255.255.255.0 skipassource=true

Neevia Document Converter Pro 6 Windows Service Blocking/freezing not converting XL/XLS/XLSX files

If you are having problems converting certain MS Office documents while running the converter as a windows service then read the below:

If you run DC Pro as a windows service try this: on the machine where this happens manually open MsExcel click on the Office Button and go to Word Options and select Popular. In the section, Personalize your copy of Microsoft Office, fill in the values for User name and Initials and reboot.

Another thing you should consider in case you run DC Pro as a service and still have problems log in with the account used by dcStart and launch (open/close) all the applications (MsWord, Excel… etc) used as parsers by DC Pro to have their registry profiles properly created.

Also make sure that MsOffice parsers run in the same user account as DCStart.

To do that type dcomcnfg in the command prompt hit enter, expand the Component Services group, expand the Computers group, expand the My Computer group, expand the DCOM Config group, find and select Microsoft Excel Document (valid for other parsers too) -> right mouse click -> Properties.

- click the Identity tab. Check the This user checkbox, press Browse and specify the Administrator account or the account used by dcStart;

- enter and re-enter the password;

- click the Security tab. Check Use custom access permissions, press Edit and add the "Everyone" user account;

- Check the Use custom launch permissions checkbox, press

- Edit and add the "Everyone" user account;

- reboot the computer;

MS SQL 2005 Debugging Stored Procedure Performance by printing current time

When you have a very large Stored Procedure which is made up of multiple queries and is slow or is taking a very long tim eto execute, you can place the following in before and after each section so that when you run it through the MS SQL management studio in the messages tab you can see exactly how long each section of the Stored Procedure takes to execute.

print 1

print convert(varchar(10),CURRENT_TIMESTAMP,114)

select ....

print 2

print convert(varchar(10),CURRENT_TIMESTAMP,114) 

select ....

print3

print convert(varchar(10),CURRENT_TIMESTAMP,114)

converting certificate from IIS to Apache

If you have a pfx file for IIS web server but you need the relevant key and certificate files for Apache web server the following will be very usefull:

# Export the private key file from the pfx file
openssl pkcs12 -in filename.pfx -nocerts -out key.pem
# Export the certificate file from the pfx file
openssl pkcs12 -in filename.pfx -clcerts -nokeys -out cert.pem
# This removes the passphrase from the private key so Apache won't
# prompt you for your passphase when it starts
openssl rsa -in key.pem -out server.key

samba jcifs single server/host multiple computer or host names

I was using jcifs in a java application to access files on a remote file server.

Samba and hence jcifs do not like it when you use a path which contains an alias rather than the actual computer or host name.

So for example if the hostname of your remote file server is fileserver1, but for some reason you want to access some files on this server using JCIFS or samba but want to refer to it as fileserver2 and in your local hosts file you have fileserver2 pointing to the same IP address as fileserver1, then when you try to access \\fileserver2\.... you get the following error:

jcifs.smb.SmbException: Failed to connect to server
java.net.UnknownHostException: fileserver2
at jcifs.UniAddress.getAllByName(UniAddress.java:315)
at jcifs.smb.SmbFile.getFirstAddress(SmbFile.java:852)
at jcifs.smb.SmbFile.connect(SmbFile.java:939)
at jcifs.smb.SmbFile.connect0(SmbFile.java:868)
at jcifs.smb.SmbFile.queryPath(SmbFile.java:1321)

The solution is the following:

1. open registry editor on the fileserver. 
2. go to HKEY_Local_Machine \ System \ CurrentControlSet \ Services \ LanmanServer \ Parameters 
3. create a new multi-string value parameter with the name of "OptionalNames". 
4. edit the data of this newly created field and add as many aliases as you want. One alias on a new line. So in the above case, you would just add fileserver2. 
5. Now on both the fileserver and your local client add fileserver2 to your hosts file (C:\WINDOWS\system32\drivers\etc\hosts) with the same IP address of fileserver1. 
6. reboot the fileserver. 

I recently found that an easier way is just to create a new value under HKEY_Local_Machine \ System \ CurrentControlSet \ Services \ LanmanServer \ Parameters on the fileserver and then reboot the fileserver:

Value name: DisableStrictNameChecking
Data type: REG_DWORD
Radix: Decimal
Value: 1

Rsync over ssh hanging / freezing in Cygwin problem

Sometimes when using rsync over ssh under cygwin in windows to synchronise files from one server to another, the rsync process copies a few files and then just hangs. Sometimes the rsync process utilises 100% cpu and sometimes 0% cpu, but in either case no more files are synchronised.

The solutions is to use rsync as a daemon on the remote server.

run the following in cygwin on the remote windows machine.

cygrunsrv --install "rsyncd" --path /usr/bin/rsync --args "--daemon --no-detach" --desc "Starts a rsync daemon for accepting incoming rsync connections" --disp "Rsync Daemon" --type auto

then create the following file /etc/rsyncd.conf

In this file you should have something like:

max connections = 10

log file = /var/log/rsync.log

timeout = 300

[pub]

comment = Random things available for download

path = /path/to/my/public/share

read only = no

list = yes

Note: in the above example the name of the module is pub.

Then start the service. You can use "net start rsyncd" to do this.

Now when you want to rsync use:

rsync ... src rsync://user@host/pub/datafolder

notice that you must use modules when using rsync daemon. In this case the module is pub.

The above rsync will copy the files to /path/to/my/public/share/datafolder

Finding all hosts on a network subnet (Windows Cygwin Bash)

Just run the below in Cygwin bash command shell:

for ip in $(seq 1 254); do ping -w 20 -n 1 192.168.1.$ip>/dev/null; [ $? -eq 0 ] && echo "192.168.1.$ip UP" || : ; done

What queries are currently running MS SQL 2005

This is the script which I always had in my archive. Following script find out which are the queries running currently on your server.

SELECT
sqltext.TEXT,
req.session_id,
req.status,
req.command,
req.cpu_time,
req.total_elapsed_time
FROM sys.dm_exec_requests req
CROSS APPLY sys.dm_exec_sql_text(sql_handle) AS sqltext

While running above query if you find any query which is running for long time it can be killed using following command.

KILL [session_id]

Email Alert when number of established connections in Apache reach a limit on Windows

This method of alerting is actually relevant for any kind of connection checking.

You can see from the below how easily the script can be customised.

I have Apache 2.2.14 mod ssl installed on a Windows Server 2003.

On this server I have also installed cygwin.

We host a number of HTTPS web applications on this server.

Recently I noticed that suddenly Apache would become totally unresponsive.

After a lot of investigation I found that if I did "netstat -nt" on the server when Apache is unresponsive, then I would see a very large number of connections in ESTABLISHED state on port 443 all from the same IP address.

When this happens Apache become unresponsive to all new requests for 5 minutes. Then suddenly all these connections are released and Apache begins to server requests again.

Now I am still investigating as to why certain valid clients seem to suddenly make hundreds of HTTPS connections to our servers and hold the connections until the Apache timeout of 300 seconds kicks in.

In conf/extra/httpd-default.com the default value for Timeout is set to 300 seconds.

Now you should not really change this value but if you are having the same problem as me you do not really want the Apache server to be unresponsive for 5 miutes (300 seconds). I have set this value to 30 seconds and at least now the server will become unresponsive for 30 seconds then it recovers.

At the same time I wanted to be alerted by email when this situation occurred.

So I did the following on the Apache server:

Download bmail from this link, extract and copy bmail.exe in to d:\tools\bin\bmail.exe

In the same directory create a bash script and call it connectionCheck.sh

assuming
1. your email address is techsupport@abc.com
2. your server is called XYZ
3. you would like to be alerted as soon as the total number of concurrent established HTTPS connections (443) reaches 50
4. your email servers IP address is 192.168.1.10 and that you can connect to this server and send emails from your Apache sevrer.

then the contents of this script looks like:

#!/bin/bash

netstat -nt grep -i established grep -i :443 >netstat.log
X=`cat netstat.log wc -l`
if [ "$X" -gt "50" ];
then
Y=`cat netstat.log`
/cygdrive/d/tools/bin/bmail.exe -s 192.168.1.10 -t techsupport@abc.com -f XYZ@abc.com -a "XYZ more than 50 https connections" -b "$Y"
fi

I also set
chmod +x connectionCheck.sh

and used vim to
:set ff=unix

To test the above so far you can change the value from 50 to 0 and then just execute the script from cygwin shell when you know there is atleast some activity on your HTTPS Apache server:

./connectionCheck.sh

This should send you an email which contains the netstat details at the time of execution.

Then create a windows scheduled task which starts from 00:00 and ends 23:59 and runs every minute and runs:

c:\cygwin\bin\run.exe bash -wait --login -c "/cygdrive/d/tools/bin/connectionCheck.sh"

Now every minute, this script will run silently.

It will send you an email everytime the established number of https connections to your apapche server goes above 50. You can change this number to whatever you want, but I think apache has a limit of 150 and if you are even hitting 100 concurrent established connections, you probably have a very busy site, unless your apache server takes a very long time to respond to each client request.

CYGWIN SSHD service not starting on Windows server 2008 due to Netlogon service not starting

I installed Cygwin with Open ssh on a Windows server 2008.

I ran the ssh-host-config as normal and configured sshd as a service.

When I tried to start the service it game me an error 1068. Something abot not being able to start sshd because it depends on the netlogon service.

I tried to start the netlogon service manually to see if that would help, but I could not start it as it started then stopped immediately saying that because the computer is part of a workgroup and not a domain then the netlogon service does not need to run!!!!

I found the solution was to change the startup type of the netlogon service from manual to automatic. Then reboot the server and everything starts to work!!!
I do not know why this works but it does for me.

https on THOMSON TG585v7 (solution)

in order to enable https tcp 443 redirection on the THOMSON TG585v7 router you need to do the following:

1. login via gui and create new administrator account.
2. make sure the default ip address of 192.168.1.254 is still valid for the device.
3. telnet to the device and login with the new admin account you just created.
4. run the following command:
service system ifdelete name=HTTPs group=wan
5. save changes by running:
saveall
6. Then setup the Game and application redirects as you would normally if you have not done so already.

changing frequency at which windows synchronises with internet time server

edit the following key in registry

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpClient

double click on 'SpecialPollInterval' in the right hand pane - default is 604800 (7 days) in seconds (Decimal). You can change the Decimal value to what ever you want. I have mine set to 14400 (4 Hours).

You will need to restart your system for this change to take effect.

Clone a Hyper-V virtual machine without SCVMM

Original post: http://tech.in4mr.com/2009/09/clone-hyper-v-vm-without-scvmm.html

Clone a Hyper-V virtual machine without SCVMM

The following assumes that you have created a machine that you would like to use as a template.

**Sysprepping this machine is not required, but if you will be making more than one copy I highly recommend it (sysprep instructions follow post).

1. Right click on the template machine in Hyper-V Manager and choose Export (Do Not Start a Template)

2. Choose the \ directory for a destination

3. Wait for the export to complete

4. Move the files and folders under \\__ up one level to \

5. Delete the folder \\__

6. In Hyper-V Manager choose “Import a Virtual Machine” from the Action menu on the top right

7. Import from the folder \

8. You will now see two <_template_name_> Virtual Machines in Hyper-V Manager

9. Right click the bottom one (it will have the most recent date) and rename it to your

10. Right click the Virtual Machine and choose “Settings” from the context menu

11. On the left hand side click on the Hard Drive named (it should be under IDE Controller 0) ** image A**

12. Click Remove

13. Repeat steps 11 and 12 for all hard disks

14. Close Settings

15. Open Windows Explorer and Navigate to \\Virtual Hard Disks

16. Rename .vhd to .vhd

17. Repeat step 15 for all remaining hard disks

18. Close Windows Explorer

19. In Hyper-V Manager right click the Virtual Machine and choose “Settings” from the context menu

20. On the left hand side click on IDE Controller 0 and select “Hard Disk” and click “ADD”

21. Repeat for remaining Hard Disks

a. If possible attach these to a SCSI Controller (this may have to be added via “Add Hardware” at the top of the page)

22. Close Settings

23. Right click the Virtual Machine and choose “Start”

24. Right click the Virtual Machine and choose “Connect”

If your template machine had Sysprep run on it:

25. The Virtual Machine’s screen will be grey for a while and then show what appears to be a Windows Server Setup Screen

26. Choose next, Accept the Eula and Enter when you are prompted for a machine name

27. Click next and the system will load for a while and then present you with the logon screen

Sysprep: http://technet.microsoft.com/en-us/library/cc766514(WS.10).aspx

Gotchas?: Make sure to read the links below if your going to sysprep a machine with Sql Server or Hyper-V
Sysprep a Sql Server: http://support.microsoft.com/kb/303774
Sysprep a Hyper-V Server: Http://blogs.msdn.com/mikekol/archive/2008/03/25/hyper-v-installation-tricks-part-1-sysprep-and-hyper-v.aspx

Rsync slow in cygwin windows 2003 server

If you are using rsync to push content from a local server to a remote server, I found it to be much faster to perform the rsync over ssh rather than using network paths.

so instead of doing something like:

rsync -avz /cygdrive/c/data/* //192.168.1.20/data/

do this:

rsync -avz /cygdrive/c/data/* -e ssh someuser@192.168.1.20:/cygdrive/c/data/

I have no idea why it is faster.

Also I found rsync over ssh to be more stable when pushing content from local server to remote server rather than pulling content from remote server to local server.

Making mirror server the principal server MS SQL 2005

In MS SQL 2005 mirroring, when the principal server fails you need to execute the following on the mirror server in order to make the mirror server active.

ALTER DATABASE [database name] SET PARTNER OFF
go

RESTORE DATABASE [database name] WITH RECOVERY
go

Shrinking MS SQL 2005 log files

If database is mirrored then remove mirroring first.

DBCC SHRINKFILE ('db log name', [MB])
go
BACKUP LOG [dbname] TO DISK = 'null'
go
DBCC SHRINKFILE ('db log name', [MB])
Go

Outlook 2007 High CPU Usage Problem - Solved

Original solution was found at:

http://weblogs.asp.net/guybarrette/archive/2008/01/13/outlook-high-cpu-usage-solved.aspx

I was having some performance issues with Outlook 2007. When running idle, it was consuming around 20-30% CPU. I know that Outlook is no small app but 20-30% CPU when idle, c'mon!
Trying to figure out what was wrong, I made sure I downloaded all the performance patches and I applied Office SP1 but still experienced a high CPU usage.
One day, I had the idea to disable all the non essential add-ins. Looks like with every software application comes with an Outlook add-in. Why do I need a SnagIt add-ins in Outlook? Why do I need a MindManager add-in? Nuisance if you ask me.
Guess what? After disabling these add-ins, Outlook CPU usage dropped to 0% with peaks at 2-4%!!!
So how do you disable Outlook ad-ins? In the Tools menu, select Trust Center.

Click on the Add-ins tab on the left of the screen and in the lower portion, select Com Add-ins from the drop down and click Go.

You can then uncheck non essential add-ins.

Every time I run Microsoft word or XL it performs installation.

I just uninstalled all instances of office from my PC and reinstalled it. This seemed to work.

MS SQL 2005 mirroring howto

My experience so far has been with setting up MS SQL 2005 mirroring under windows workgroups (not domains). The SQL servers did not have FQDNs. The database I was trying to setup was configured with "simple" recovery model.

Although the following worked for me, not every step may be necessary and the below is not necessarily the recommended method of achieving mirroring or setting up SQL Server:

1. I had the latest service pack for SQL server 2005 on both servers.

2. Modify the recovery model to "FULL". Then I modified a couple of records in the database to make sure there is something in the transaction logs.

3. Perform a full backup of principal and restored on mirror server.

4. Perform a transaction log backup and restore onto mirror server.

5. Both SQL server and the SQL Server Agent services on both the principal and mirror server are running as the same windows user with the same password. I made this windows user part of the Administrators group on both servers.

6. On the last page of the mirroring configuration wizard, click on "do not start mirroring". This will take you to the parent window where you can replace the Server Network Address for both your principal and mirror servers with the IP address rather than database name.

7. Everything else was either very obvious or I left as default.

Once I did the above tasks, then I clicked on start mirroring and everything just started working.