PowerShell For Fun

Recently, I was working on a bioinformatics research project which needed to download hundreds of gene mRNA sequences. I have all the gene IDs in one text file. So a simple PowerShell Script could solve my problem.

I have a old post talking about NCBI Entrez eUtils tools. Today, I will use the eFetch tool (also included in eUtils). The script is simple and self-explaining.

# ===========================================================================
#
# Author:      Tony (http://MSHForFun.blogspot.com)
# File:        Efetch.ps1
# Description: Download gene sequences using NCBI eUtils.eFetch tool
# Reference: http://eutils.ncbi.nlm.nih.gov/entrez/query/static/eutils_example.pl
# Reference: http://eutils.ncbi.nlm.nih.gov/entrez/query/static/efetch_help.html
# Reference: http://eutils.ncbi.nlm.nih.gov/entrez/query/static/efetchseq_help.html
# 
# ===========================================================================
param
(
  [string] $Path=$(throw "Please Specify a file")
)
$BaseURL = "http://www.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=nucleotide&id="
$Option= "&rettype=fasta&retmode=text"
$WebClient = new-object System.Net.WebClient
$SavePath = $Path + ".result"
if (test-path $savePath)
{
  del $SavePath
}
foreach ( $id in (get-content $path))
{
  # Construct eFetch URL
  $URL=$BaseURL + $id + $Option
  Write-Progress -Activity "Download Sequences" -Status "Submit gene $Id"
  # Submit and download data
  $Data = $WebClient.DownloadString($URL)
  # Parse Data
  if ($Data.Length -gt 1)
  {
    Write-Progress -Activity "Download Sequences" -Status "$id OK"
    # Write to Console
    $data
    # Wrtie To file
    $data >> $SavePath
  }
  else
  {
    Write-Progress -Activity "Download Sequences" -Status "$Id is not found!"
    "$Id is not found!`n`r"
    "$Id is not found!`n`r" >> $SavePath
  }
  # Try not to overload NCBI Server
  start-sleep 1
}
# Clear Progress pane
Write-Progress -Activity "Download Sequences" -Status "Done" -completed

You need a text file (genes.txt) to test this script:

0
NM_008176
NM_009140
NM_009141
NM_011333
NM_013654
NM_016960
NM_009142
NM_008491
NM_031168
NM_009883
NM_007679
NM_010030
NM_009971
NM_010809
NM_008607
NM_030612
NM_011198
NM_007987

If you are a biologist, you can see what kind of genes I am intersted in. The first "0" is just to cause an "Not Found" Error. You can run this script like following:

.\efetch.ps1 genes.txt

Your results is printed to screen as well as "genes.txt.result" file.

Have Fun

Tags: msh    monad   PowerShell

Any remote shell application is dangerous because you open a door to outside world. So does PowerShell Remoting. Previous version of PowerShell Remoting totally depended on NegotiateStream. You have no control of login process or you can not limit user access.  So anyone with a valid local or domain account can login remotely into your computer. As more and more people began to download and install PowerShell Remoting on their computer, I decided to add client access control policy to PowerShell Remoting.

Firstly, you should limit maximum number of clients which can connect to server simultaneously. This is done by modify registry:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PowerShellRemoting\Parameters]
"MaxClient"=dword:0000000a

Default value is 10. Given 256MB memory is standard configuration for desktop PC, this number is more than enough if you decided to login to your desktop from home network. You can definitely increase this number if you have more clients but remember each runspace will allocate quite a few of memory. So do some experiment and calculation then you can find a reasonable number.

Secondly, you should limit maxium number of clients which can connect to server simultaneously from Same IP address. This is done by modify registry:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PowerShellRemoting\Parameters]
"MaxClientPerIP"=dword:00000002

Default value is 2. You can also change that but you don't want all your available slots of connection were occupied by clients from same computer.

Thirdly, you may want to limit which user can login from which IP address. This is done by modify
%Program Files%\PowerShell Remoting\user.xml
Default installation only allow user in Administrators group login from localhost. So you have to have your own user.xml before you can login from another IP. If user.xml file parse error occorred when start service, default policy will be loaded. If you can't login, check log file or EventLog for details of parse error.
User.xml file contain information about your user access control list. It has SEVEN tags:

1. <NetworkACL>: container of one or more access control entry (<NetworkACE>)
2. <NetworkACE>: access control entry container of <Account>, <Access> and <IPRange>
3. <Account>: Valid Group or User
4. <Access>:  Allow / Deny
5. <IPRange>: container of <IP> and <Subnet>
6. <IP>: IPAddress. 0.0.0.0 for any IP, 127.0.0.1 for localhost, 192.168.0.2 etc. IPv6 string should be fine, but I have not been able to test it.
7. <Subnet>: IPv4 subnet. 192.168.0.0/255.255.255.0 or CIDR like format 192.168.0.0/24.

Within <NetworkACE> and </NetworkACE> tags, order matters.

1. <Account>
2. <Access>
3. <IPRange>

Within <IPRange> and </IPRange> tags, <IP> and <Subnet> can be used in any order.
As you can probably already figured it out : After user provide credential and got a token (WindowsIdentity), PowerShell Remoting will try to match User/Group and IP range in records of access control list. If no record was found, access is denied. If a records match User/Group and IP range were found, PowerShell Remoting will assess access in following order:

User Deny > User Allow > Group Deny > Group Allow

For example: if client were a privileged user (Administrators group) from 192.168.0.2, but there were only one record which wanted to deny access of Users group from subnet 192.168.0.0/255.255.255.0. Then access from this client will be denied because any user belongs to Administrators group also belongs to Users group and 192.168.0.2 belongs to subnet 192.168.0.0/255.255.255.0.

After you logged in and get a PowerShell prompt, you can check $UserACL for serialized access control list.

Right now, subnet parsing and match could be buggy. So tell me if you found it did not act as expected.

These changes could make it more difficult to start using PowerShell Remoting. But I believed that you will like them later on. Remember, if you make any changes to previous setting, you will have to restart service to make them take effect.

Stop-Service PowerShellRemoting
Start-Service PowerShellRemoting

There is also some exciting new feature:

1. PowerShell Remoting Client becomes a PSSnapin now. So installation become usier. Just run install.ps1 in client folder and invoke Start-RemoteHost, you are on you way to your remote shell.
2. Ctrl+C Ctrl+Break support.
   
Try

dir c:\windows\system32


And press Ctrl+C to cancel it. If you don't want to handle Ctrl+C, you can set


$RemotingClient.CanHandleCancelKey = $false


But you probably do not want to do that.
Ctrl+Break will also quit PowerShell Session. So Use it with caution.


3. You will find out that information written in log file has been dramatically reduced because I am going to swith to EventLog in later verion.

I can actually run PowerShell Remoting Client with in PowerShell Analyzer. But its ReadLine popup window really bothers me.

Tags: msh    monad   PowerShell

Version 0.2.5.1

Uninstall old version of PowerShell Remoting before install newer version.

1. Support Ctrl+C and Ctrl+Break to cancel current pipeline (UDP datagram, So not 100% reliable)


2. Support Maximum Client per IP option (default 2)


3. Customized client filter policy (%Program Files%\PowerShell Remoting\user.xml, see sample files for format information)

Important: default installation only allow user in Administrators group login from localhost. So you have to have your own user.xml before you can login from another IP. If user.xml file parse error occorred when start service, default policy will be loaded. Check log file or EventLog for details of parse error.


4. $UserACL variable for current client filter policy


5. Client is installed as PSSnapin


6. New install/uninstall script for both server and client, new start-remotehost.ps1 script


7. CanThrowException and CanHandleCancelKey property of client (For developer)


8. Change Log file path to %Documents and Settings%\NetworkService\Local Settings\Application Data\PowerShellRemoting.log


9. Change Service Name to "PowerShellRemoting"


10. Service related Exception is also Logged to Eventlog (EventLog: Application, source: PowershellRemoting)


11. Other Bugs fixed

Have Fun

Tags: msh    monad   PowerShell

Windows PowerShell is powerful (hehe, sounds like a bad salesman). To make it even better, user can extent it by PSSnapin. PSSnapin can contain PSProviders, Cmdlets and other class library. But there is a catch. PSSnapin is arbitrary code which could be bad designed, bugs loaded and somewhat out of your control. If you are not careful when install and execute, PSSnapin can cause serious security problems. Let's look following example:

Supposed you have a PSSnapin:

using System;
using System.ComponentModel;
using System.Management.Automation;
namespace TestSnapin
{
    [RunInstaller(true)]
    public class MySnapin : PSSnapIn
    {
         public override string Name
         {
            get { return "Test"; }
         }
        public override string Vendor
        {
             get { return "http://MSHForFun.blogspot.com/"; }
        }
        public override string Description
        {
            get { return "Test"; }
        }
    }

   public class Class2
   {
        public string Who
        {
              get { return "Good Guy!"; }
        }
   }
}

You built it by yourself or simply downloaded it from internet. You (An Administrator) copied it to a folder and ran installutil.exe to install it. Then you tried to use it in PowerShell:

>get-pssnapin -reg

Name : Test
PSVersion : 1.0
Description : Test

> add-pssnapin test
> $a = new-object TestSnapin.Class2
> $a.Who
Good Guy!

Looks perfect, right? But

1. The folder where you save your PSSnapin is writeable by any user, so anyone can change your PSSnapin.
2. Your PSSnapin assembly is not signed, so PowerShell will load it without checking its integrity. See the registry key about your PSSnapin

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\Test]
"PowerShellVersion"="1.0"
"Vendor"="http://MSHForFun.blogspot.com/"
"Description"="Test"
"Version"="1.0.0.0"
"ApplicationBase"="D:\\ps1"
"AssemblyName"="Snapin1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"
"ModuleName"="D:\\ps1\\snapin1.dll"

Suppose there were a malicious user wrote a PSSnapin like yours PSSnapin except Class2:

public class Class2
{
    public string Who
    {
        get { return "Bad Guy!"; }
    }
}

He just overwrote your PSSnapin with his DLL. What happened next?

> add-pssnapin test
> $a = new-object TestSnapin.Class2
> $a.Who
Bad Guy!

That is to say user can run any code he wanted. In a worst condition, when you (An Administrator) tried to use this PSSnapin, you were actually tricked into running malicious code as Administrator!

So Lessons:

1. Keep an eye on your PSSnapin. Never put them in a directory which is writeable by unprivileged user
Edit: 2006-05-16 08:36
"Since snapins are programs that you install, it is wise to apply the tenets and best practices of software installation to them...The potential security problems don't come from PSSnapins -- they come from executing code that you don't trust. " --Lee
Edit: 2006-05-16 08:36

2. Always use a Signed PSSnapin. If a signed PSSnapin were altered, PowerShell will not load it.

> add-pssnapin test
Add-PSSnapin : Cannot load PSSnapIn test. Encountered following error: PSSnapin
module D:\ps1\snapin1.dll doesn't have required PSSnapin strong name Snapin1,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=3daf9f8a713aaa33.
At line:1 char:13
+ add-pssnapin <<<< test

Tags: msh    monad   PowerShell

I was reading post on "The Old New Thing" titled "A new scripting language doesn't solve everything" yesterday. I was just going to write something about this. Lee was one step ahead of me and did a response post titled "Nothing solves everything - PowerShell and other technologies".

Monad (well, I mean PowerShell) is cool. We have new shell and new script language. But it is still going to be a very long time before we can do everything using cmdlet because not only we need time to write new cmdlets, but also we are limited by what .NET Framwork can do. Most important, monad is NOT supposed to "solve everything". Instead monad is supposed to work cooperatively with "OLD" techniques. So we have to deal with backwards compatibility issue. For example, other scripting language, unmanaged code and legacy program (console application). I will talk about support for console application in monad hosting application today.

For a local interactive user, powershell.exe provides nice support for "old" console application inherited from cmd.exe. Common user don't even feel much difference between cmd.exe and powershell.exe if they just invoke

ping localhost

For a programmer, there is a difference lying underneath the surface. Monad using PSHostUserinterface and PSHostRawUserinterface to get user input and display results if user invokes cmdlets. To support legacy console application, powershell.exe actually create a new process (outside of pipeline) and redirects stdin/stdout/stderr of legacy console application to its own console.

PowerShell.exe works fine for two reasons:
1. PowerShell.exe itself is a console application.
2. PowerShell.exe does NOT care whether the legacy console application runs in a different process or not. (cmdlet runs in-process)

It becomes a nightmare if you are going to write a hosting application. Considering what will happen when user invokes console application inside your hosting application like "netsh.exe" which requires user input/output and you did not redirect input and output.
1. Legacy console application start a new process which is out of your control
2. If your hosting application is not a console application (like PowerShell Analyzer), legacy console application start a new Console window which is out of your control. "You can loose output or hang waiting for input that never comes." -- William Stacey [MVP]
3. For a remoting host application (like my PowerShell Remoting), it is even worse because the new console window is on another computer, there is no way for remote user move their cursor to the console window and provide input.

It is NOT a trivial work to solve this problem. My "PowerShell Remoting" does not solve this issue (A NotSupportException will be thrown if user invoked legacy console application). Karl Prosser's PowerShell Analyzer  does not solve this problem. If you follow the disscussion in newsgroup: Redirect msh.exe in/out  you will find more people are struggling with this issue.

So here comes the question: Should this issue been taken care of by internal host or external hosting application?

I think the answer is both YES and NO.
1. NO: The Design of Monad internal host is to focus on process scripting language, work with objects and piplines. External host is supposed to take care of application logic and user interface.
2. YES: legacy console application is a PROBLEM for ALL monad hosting application. So monad should provide more support for it.

For me, a perfect solution would be that all legacy console application use PSHostUserinterface and PSHostRawUserinterface for ALL output/input (There are protential big problem behind this). Maybe improvement for PSHostUserinterface and PSHostRawUserinterface's definition is needed. NotifyBeginApplication() and NotifyEndApplication() is not good enough.


Have Fun

Tags: msh    monad   PowerShell

What's New

Version 0.1.1.7
1. Recompiled for new Windows PowerShell RC1 (Refresh version)
2. Using "Thread Pooling" method to schedule multiple Host threads. So Sever can potentially
accept more connections. (Old version uses one thread per connection method).  
3. Using a separate thread actively reclaim resources from broken connection and dead host.
4. Clean exit when stop service: Disconnect all clients and dispose all running hosts.
5. Gracefully disconnect client when server reach maximum client capacity.
6. Fix: Nested Prompt stack error when multiple clients connected.
7. Fix: Server unable to exit when connection closed unexpectedly by client.
8. Fix: "SetShouldExit" method re-throw "SocketException" Error

Tags: msh    monad   PowerShell

Version 0.1.0.317
1. NestedPrompt (suspend host)
2. Multiple line input mode
3. Use local UI to get login credential
4. Save RawUI state on start, and Reset RawUI state on exit.
5. $CurrentUser Variable (WindowsIdentity Object represent current
login user)
6. Load user profile in following order
1) "\Documents and settings\All users\PsConfiguration\profile.ps1"
2) "\Documents and settings\All users\PsConfiguration\PowerShellRemoting_profile.ps1"
3) "My Documents\PsConfiguration\profile.ps1"
4) "My Documents\PsConfiguration\PowerShellRemoting_profile.ps1"

Tags: msh    monad   PowerShell

What Can Tweaking Your Prompt Do For You?

Windows PowerShell have a default prompt in one color (usually gray) that tells you your current working directory. This is OK, but you can do much more with the prompt.
1. All sorts of information can be displayed (machine name, host name, user name, time & date ...)
2. The prompt can use colors
3. You can also manipulate the windows title to dispaly more information.
4. Other RawUI operation is allowed (move cursor)

Why Bother?

Beyond looking cool, it's often useful to keep track of system information.
1. Get your current working directory (how many files in current directory)
2. Current system time, how many process is running
2. If you use my su.msh, you would like to know you current windows identity
3. If you use my powershell remoting, different color helps distingish between local host and remote host.
4. Colorizing your prompt is the ability to quickly spot the prompt when you use scroll console.

First step: Profile and prompt function

Windows PowerShell use Profile to customize user environment. For more detail, about profile
The Story Behind the Naming and Location of PowerShell Profiles:
http://www.leeholmes.com/blog/TheStoryBehindTheNamingAndLocationOfPowerShellProfiles.aspx

1. You can have different prompt for different shell (Maybe you use makeshell.exe generated you own shell)
2. If you have different prompt function defined in multiple profile, the last one excuted take effect
Remember the excution order is

1. "All users" profile is loaded from "<Installation Directory>\profile.ps1"
   
2. "All users," host-specific profile is loaded from "<Installation Directory>\Microsoft.PowerShell_profile.ps1"
   
3. Current user profile is loaded from "<My Documents>\WindowsPowerShell\profile.ps1"
   
4. Current User, host-specific profile is loaded from "<My Documents>\WindowsPowerShell\Microsoft.PowerShell_profile.ps1"

Edit 09-29-2006

Windows PowerShell RC2 changed profile location to

<My documents>\WindowsPowerShell
Now you can customize you prompt in you profile (say <my document>\psconfigurtion\profile.ps1)
this is typical prompt function looks like:

function prompt
{
    "PS " + $(get-location) + "> "
}

You can do whatever you want in prompt function. But remember
1) Always return a [string], otherwise Windows PowerShell will use default "PS> " prompt.
2) Try to limit your prompt in one (short) line
3) Host will evaluate prompt frequently, so don't do crazy stuff to slow down your work.

Colorized prompt

function prompt
{
    Write-Host ("PS " + $(get-location) +">") -nonewline -foregroundcolor Magenta
    return " "
}

Random color

function prompt
{
    $random = new-object random
    $color=[System.ConsoleColor]$random.next(1,16)
    Write-Host ("PS " + $(get-location) +">") -nonewline -foregroundcolor $color
    return " "
}

Cursor Movement
Display current  time at the end of prompt line (this will mess up you console buffer)

function prompt
{
    $oldposition = $host.ui.rawui.CursorPosition
    $Endline = $oldposition
    $Endline.X+=60
    $host.ui.rawui.CursorPosition = $Endline
    Write-Host $(get-date).Tostring("yyyy-MM-dd HH:mm:ss")
    $host.ui.rawui.CursorPosition = $oldposition
    Write-Host ("PS " + $(get-location) +">") -nonewline -foregroundcolor Magenta
    return " "
}

Use Window Title
Show current user, host, current line number

$global:CurrentUser = [System.Security.Principal.WindowsIdentity]::GetCurrent()
function prompt
{
    $host.ui.rawui.WindowTitle = $CurrentUser.Name + " " + $Host.Name + " " + $Host.Version + " Line: " + $host.UI.RawUI.CursorPosition.Y
    Write-Host ("PS " + $(get-location) +">") -nonewline -foregroundcolor Magenta
    return " "
}

Make some noise
if your command  take very long time to run, beep when it is done.

function prompt
{
    Write-Host ("PS " + $(get-location) +">") -nonewline -foregroundcolor Magenta
    return "`a "
}

More information
Count number of files(or items) in current path and number of process running

function prompt
{
    $host.ui.rawui.WindowTitle = "Files: " + (get-childitem).count + " Process: " + (get-process).count
    Write-Host ("PS " + $(get-location) +">") -nonewline -foregroundcolor Magenta
    return " "
}

Something for readers:
1. laptop battery is low ! (Edit 2006-07-21: Already done by Musings of a PC)
2. You got new mail !
3. LAN cable disconnected !

Have Fun

Reference:
http://www.tldp.org/HOWTO/Bash-Prompt-HOWTO/

Tags: msh    monad   powershell